Description:
The pipeline builds failing with cannot resolve 'some column name' given input columns error. Sample error log looks like below,
20/10/13 12:42:05 ERROR DefaultTaskDagExecutorImpl: Exception: cannot resolve '`created_yearmonth`' given input columns: [publish.dim_employee.contactid, publish.dim_employee.source_system, publish.dim_employee.id, publish.dim_employee.ziw_effective_start_timestamp, publish.dim_employee.employeenumber, publish.dim_employee.middlename, publish.dim_employee.employeewid, publish.dim_employee.ziw_status_flag, publish.dim_employee.division, publish.dim_employee.firstname, publish.dim_employee.role, publish.dim_employee.startdate, publish.dim_employee.ziw_active, publish.dim_employee.employee_name, publish.dim_employee.department, publish.dim_employee.psft_employeeid, publish.dim_employee.employee_id, publish.dim_employee.lastname, publish.dim_employee.branch, publish.dim_employee.ziw_is_deleted, publish.dim_employee.ziw_row_id, publish.dim_employee.ziw_target_timestamp, publish.dim_employee.ziw_effective_end_timestamp]; line 1 pos 376;
'Project [startdate#479, employeenumber#480, firstname#481, employee_id#482, role#483, id#484, psft_employeeid#485, contactid#486, source_system#487, employeewid#488, lastname#489, middlename#490, employee_name#491, branch#492, division#493, department#494, ziw_target_timestamp#495, ziw_status_flag#496, ziw_effective_start_timestamp#497, ziw_effective_end_timestamp#498, ziw_active#499, ziw_is_deleted#500, ziw_row_id#501, 'created_yearmonth]
Root Cause:
This happens if the metadata goes out of sync or due to the wrong pipeline design.
Solution:
The user needs to correct their pipeline design to fix this issue.
Applicable versions:
All versions of Datafoundry.