API rejects zip file upload over ":ZipUploadFilesLimit", while the web UI saves them as-is #12751

Activity

1. Shuxian-Account

   added

   Type: Buga defecta defect

   on Sep 29, 2026
2. dataversebot

   added this to IQSS Dataverse Projecton Sep 29, 2026
3. pdurbin commented on Oct 19, 2026

   

   pdurbin

   on Oct 19, 2026

   Member

   More actions

   @Shuxian-Account thanks for opening this issue. I can reproduce it on the latest in the "develop" branch (7b998a3).

   I've never seen anyone use :ZipUploadFilesLimit in this way. 😄 I didn't realize that setting it to a low number (like 1 in your case) is a work around for allowing zip files to be uploaded.

   Some screenshots:

   

   So, yes, it works, the zip file is kept intact but this is not idea. Instead we should probably offer a checkbox for people who want to skip unzipping:

   • Support uploading of archives (ZIP, other). #8029

   I also checked the API and you're right, It shows an error like this:

   {"status":"ERROR","message":"The number of files in the zip archive is over the limit (1); please upload a zip archive with fewer files, if you want them to be ingested as individual DataFiles."}

   I tested with sample-dataset.zip and used these commands:

   % export API_TOKEN=1d1d6670-ac7f-45d6-8bc3-1b9cd4c3bcf0
   export FILENAME='sample-dataset.zip'
   export SERVER_URL=http://localhost:8080
   export PERSISTENT_ID=doi:10.5072/FK2/MV8S0T
   % 
   % curl -H "X-Dataverse-key:$API_TOKEN" -X POST -F "file=@$FILENAME" -F 'jsonData={"description":"My description.","directoryLabel":"data/subdir1","categories":["Data"], "restrict":"false", "tabIngest":"false"}' "$SERVER_URL/api/datasets/:persistentId/add?persistentId=$PERSISTENT_ID"
   {"status":"ERROR","message":"The number of files in the zip archive is over the limit (1); please upload a zip archive with fewer files, if you want them to be ingested as individual DataFiles."}
   

   I also tried the SPA, the new frontend, but because it only supports S3 at the moment, zip files are not unzipped:

   

   I'm not sure what we should do with this issue, but I wanted to post my findings at least!

   I would say the API is doing what we want by rejecting the file. We COULD change the old UI (JSF) to be in line with the API (to reject the file) but we are trying to touch JSF as little a possible so we can focus on the new UI.
4. Shuxian-Account commented on Oct 20, 2026

   

   Shuxian-Account

   on Oct 20, 2026

   Author

   More actions

   Hi Phil, thanks for reproducing this and for the detailed write-up.

   One concern about aligning the UI with the API: if both reject the upload, that removes the only way we currently have to keep zip files intact. Our file store is on NFS and inodes are a real constraint — many of our datasets hold thousands of files each, and unpacking them consumes inodes very fast. Setting :ZipUploadFilesLimit to 1 is a workaround, but it’s the only one we found.

   We’d prefer the opposite alignment: have the API save the zip as-is with a warning, as the UI does.

   And if #8029 is implemented, could the opt-out cover API uploads too, and not just a checkbox in the UI? Many of our large deposits are scripted.
5. pdurbin commented on Oct 20, 2026

   

   pdurbin

   on Oct 20, 2026

   Member

   More actions

   @Shuxian-Account that's what I was afraid of, that you'd rather go the other way. 😅

   I'm not sure if we document this anywhere but our standard advice, the workaround we often give, is to double zip. That way Dataverse unzips the outer layer and the inner zip is deposited as is. Does that make sense? It's a workaround, for sure, but I think a lot of people use it (because they have to). I use it myself. This is what I do:

   zip -r primary-data.zip primary-data -x '**/.*' -x '**/__MACOSX'
   zip -r outer.zip primary-data.zip -x '**/.*' -x '**/__MACOSX'
   

   That is. The final, deposited file is called primary-data.zip. From https://github.com/pdurbin/open-source-at-harvard-primary-data
6. Shuxian-Account commented on Oct 26, 2026

   

   Shuxian-Account

   on Oct 26, 2026

   Author

   More actions

   Hi Phil, thanks for spelling it out. We’re aware of the double-zip workaround.

   The difficulty for us is that it’s a depositor-side workaround. Our repository is self-deposit: researchers can upload and publish their own datasets. We can recommend double zipping in our guidance, but we can’t force it, and we only find out what was deposited after the files are already on disk. That’s why inode consumption is an operational problem for us.

   So what we’re really after is something we can control on the server side. At the moment :ZipUploadFilesLimit is the only lever we have.
7. pdurbin commented on Oct 26, 2026

   

   pdurbin

   on Oct 26, 2026

   Last edited by pdurbin

   Member

   More actions

   Right, the only lever.

   I realized I never answered this:

   And if #8029 is implemented, could the opt-out cover API uploads too, and not just a checkbox in the UI? Many of our large deposits are scripted.

   Yes, the idea is that both UI and API would be covered. Of course, the old UI (JSF) is a bit deprecated but the new UI ( https://github.com/IQSS/dataverse-frontend ) would definitely have a checkbox.
8. Shuxian-Account commented on Oct 27, 2026

   

   Shuxian-Account

   on Oct 27, 2026

   Author

   More actions

   Thanks Phil, good to know both would be covered.

   One clarification on our side: we don’t actually need unzipping turned off completely. A threshold is fine, and :ZipUploadFilesLimit already gives us exactly that in the web UI — a zip over the limit is kept intact, and anything under it is still unpacked as usual. We set it to 1, but 10 would suit us just as well.

   This brings it back to the original report. When the limit is exceeded, the UI saves the zip as-is, as described in the User Guide, while the API rejects it. The API is the one that diverges from the documented behaviour, so could it be brought in line with the UI? That alone would solve our problem, and it doesn’t depend on #8029.

   The other half of the report is still open too: the config guide’s description of :ZipUploadFilesLimit doesn’t say that the limit controls unpacking, or that the UI and API behave differently once it’s exceeded. I’d be happy to open a PR on the guides, too.

   Thanks again.