2. ASPRunner.NET Tips and tricks
  3. Import large file in Azure Webapp
This topic is locked

Import large file in Azure Webapp

 		6/18/2024 10:51:11 PM
ASPRunner.NET Tips and tricks
A
 Arashg author

I have a large file 200k records and I need to do the import , it's taking too long in Azure and it error out with following error:

500 - The request timed out.
The web server failed to respond within the specified time.

Is there anything any way to do that?

I have already added these and it did help for the previous errors, but still could not do anything to make it faster or pass the " 500 - The request timed out." error.
<httpRuntime requestPathInvalidCharacters="" targetFramework="4.8" maxRequestLength="102400000" executionTimeout="99999999" />

...

<security>
<requestFiltering>
<requestLimits maxAllowedContentLength="205000000" /> <!-- 20.5 MB - making it match maxRequestLength to fix issue with uploading 20mb file -->
</requestFiltering>

</security>
</system.webServer>

Any help is really appreciate it.
Dalkeith 6/19/2024

It is likely that you will be limited by your host on the web app service. For that many records you could try upping your hosting level for the period of import alternatively it might be time to directly import into the database.
Sergey Kornilov admin 6/20/2024

This is looks like something that your web hosting company needs to fix/adjust for you. Talk to them.

Dalkeith 6/21/2024

So I have come across this before - I find azure web app service doesn't mind 2500 import but after that it starts timing out..
This is on the cheapest web app service plan.

I ended up writing a simple bat file which will take a csv and break it out into smaller files

Batch File CSV splitter code

@echo off

setlocal ENABLEDELAYEDEXPANSION

REM Edit this value to change the name of the file that needs splitting. Include the extension.
SET BFN=C:\csv\target.csv

REM Edit this value to change the number of lines per file.
SET LPF=2500

REM Edit this value to change the name of each short file. It will be followed by a number indicating where it is in the list.
SET SFN=SplitFile

REM Do not change beyond this line.

SET SFX=%BFN:~-3%
SET /A LineNum=0
SET /A FileNum=1
For /F "delims==" %%l in (%BFN%) Do (
SET /A LineNum+=1
echo %%l >> %SFN%!FileNum!.%SFX%
if !LineNum! EQU !LPF! (
SET /A LineNum=0
SET /A FileNum+=1

)

)

endlocal

Pause