KISHORE VELETI replied to KISHORE VELETI's discussion Sample DTL code for APACHEAVRO and PARQUET FILE FORMAT in DMX for Hadoop
"Here is the content of all_layouts.dtl

/DTL RUNTIMEVARIABLES ON       /DELIMITEDRECORDLAYOUT user_layout_input  {     /* This is definitely a comment */    userid en NOTNULLABLE, /* This is a comment */    first_name character  NOTNULLABLE  ,    la…"
Feb 12, 2016
KISHORE VELETI replied to KISHORE VELETI's discussion Sample DTL code for APACHEAVRO and PARQUET FILE FORMAT in DMX for Hadoop
"Hi All,

Here is the sample DTL I developed and I am getting compilation error for this. I understand the error but don't know how to fix it.

DMExpress : (SCHMFRM) Invalid schema format in the file "/home/dmxdemo/./all_layouts.dtl"            (AVRO…"
Feb 12, 2016
KISHORE VELETI posted a discussion in DMX for Hadoop
Dear All,Could you please a sample DTL with Apache Avro output or Parquet output file format.Basically I am looking following scenarios:INPUT could bea.) Text file or b.) Avro file or c.) Parquet file formatOUTPUT could bea.) Text file or b.) Avro f…
Feb 12, 2016
KISHORE VELETI posted a discussion in DMX for Hadoop
Hi SyncSort teamWe are evaluating this use case in SyncSort, please help us our understanding.We have a table T1_Current in Hive (Database name say current_data_db)We have another table T1_New in Hive (Database name say newdata_db)Everyday we get ne…
Feb 9, 2016
KISHORE VELETI replied to KISHORE VELETI's discussion Sample SyncSort job to copy contents of a LINUX folder to HDFS location in DMX for Hadoop
"Chris,
We are trying to connect from Windows 32-bit host to Cloudera HDFS and it is giving an error. Same thing with SyncSort Cloudera VM.
Error message says contact SyncSort support.
Here is my setup:
On Windows :
Installed SyncSort Windows client…"
Feb 9, 2016
KISHORE VELETI replied to KISHORE VELETI's discussion Sample SyncSort job to copy contents of a LINUX folder to HDFS location in DMX for Hadoop
"Hi Chris,
Thanks for all your comments for this post and the others. I will try all your recommendations, going to use SyncSort for various scenarios.

Thanks,
Kishore Veleti A.V.K."
Feb 9, 2016
KISHORE VELETI replied to KISHORE VELETI's discussion CDC job (J_FileCDC.dxj) is running locally, how to make it run on Hadoop? in DMX for Hadoop
"Chris,
Below are the logs and commands I executed. Thanks for your time.
cd /UCA/bin./prep_dmx_example.sh ALLcd /UCA/Jobs/HDFSLoad/DMXHDFSJobs-bash-4.1$ dmxjob /run J_HDFSLoad.dxjDMExpress Job : (TREXPINF) the trial will expire in 30 day(s) on Mar 5…"
Feb 4, 2016
KISHORE VELETI posted a discussion in DMX for Hadoop
Dear All,I need to develop a custom functions in SyncSort DMX-H using Java/Scala/Groovy i.e other than C / C++. Is it possible to develop, if yes could you please share the documentation for the same.Below is my requirement :Left hand side is a data…
Feb 4, 2016
KISHORE VELETI replied to KISHORE VELETI's discussion Is DTL approach appropriate for multiple files handling and CDC handling? in DMX for Hadoop
"Somehow all my text formatting of above comments is lost, sorry for the inconvenience."
Feb 4, 2016
KISHORE VELETI replied to KISHORE VELETI's discussion Is DTL approach appropriate for multiple files handling and CDC handling? in DMX for Hadoop
""
Feb 4, 2016
KISHORE VELETI posted a discussion in DMX for Hadoop
Hi All,I just found in GitHub SyncSort DTL approach and very excited about it.To handle many number of files belonging to multiple tables in Hive i.e. copy from local Linux host to HDFS based on a filter criteria I thought DTL approach might help.He…
Feb 4, 2016
KISHORE VELETI replied to KISHORE VELETI's discussion CDC job (J_FileCDC.dxj) is running locally, how to make it run on Hadoop? in DMX for Hadoop
"Chris,
I could not find any logs in /usr/local/dmexpress/logs in Test Drive machine. Where can I find the logs for the job I executed which ran successfully.

Thanks,
Kishore Veleti A.V.K."
Feb 3, 2016
KISHORE VELETI posted discussions in DMX for Hadoop
Feb 3, 2016
KISHORE VELETI replied to KISHORE VELETI's discussion CDC job (J_FileCDC.dxj) is running locally, how to make it run on Hadoop? in DMX for Hadoop
"Hi Chris,

Thanks for your quick response appreciate it.

My understanding is when we submit SyncSort jobs they will run as MapReduce jobs. Based on this I am checking the YARN for an application id and I do not see any applications being run.
Yes I…"
Feb 3, 2016
KISHORE VELETI replied to KISHORE VELETI's discussion CDC on Hadoop for multiple files in a single folder in DMX for Hadoop
"Here is a link describes high level how we can dedup in Pig
I developed similar kind of solution in the past and no of lines of code is very simple and can handle multiple tables (used Groovy language which Pig supports it)
Pig code was max 15 lines…"
Feb 2, 2016
More…