Problem Description:


Export of data to SQL Server fails with the below Azure exception when Infoworks is running on Azure.


ERROR] 2019-10-07 08:57:57,648 [main] infoworks.discovery.export.Driver:196 :: java.util.concurrent.ExecutionException: infoworks.tools.ExceptionHandling.IWException: org.apache.hadoop.fs.azure.AzureException: java.lang.IllegalArgumentException: StreamWriteSizeInBytes
at java.util.concurrent.FutureTask.report(FutureTask.java:122)


Root cause:


This is an exception coming from the Azure end but not from Infoworks end. During the full export job to SQL server DB, Infoworks EDO2 calls a delete function to delete the files in the Azure file system from Infoworks end and that delete function again makes some azure native file system calls.


In one of these azure file system functions calls, azure is trying to set the stream bytes and it is getting this value from the constant fs.azure.write.request.size set at the HDFS level.


Azure code is checking if this file block size is less than 16KB or greater than 4MB (included), then it is throwing the illegal argument exception.


public void setStreamWriteSizeInBytes(final int streamWriteSizeInBytes) {
if (streamWriteSizeInBytes > Constants.MAX_BLOCK_SIZE || streamWriteSizeInBytes < 16 * Constants.KB) {
throw new IllegalArgumentException("StreamWriteSizeInBytes");


Solution:


Perform the below steps to resolve this issue.

a) Set fs.azure.write.request.size to 3994304 (slightly less than 4MB). The default value for this configuration is 4 MB. 

b) Restart the HDFS service from Ambari if it asks for a service restart (If you see a yellow notification asking for service restart).

e) Run the export job again.


Applicable Infoworks EDO2 Versions.


v2.8.x