0

Workspace ! Not being recovered where images are held & deleted.

Storage of image files and deleting them do not automatically give you your  workspace back.

I have copied this from another post that I answered as it seems to me that other users are now having issues with their database bloating regarding images. ( I can only speak here about the cloud version - I have no knowledge of other versions)

Like other users I became aware of my DB's being bloated and preventing backs etc being done  or not getting any space back after deleting images etc.

So having spend many hours back and forth with Ninox support and this is my findings.

Holding images in Image fields then manually deleting them by clicking the 3 bars on the image and selecting delete image. This removes the image from that record (Soft Deletes it - explained further below)

Removing the same field Images via script ie xfield := null or xfield = " " - again only TEMP Soft Deletes it from the image field However it still attached to the currrent record by being added to paperclip as an attachment ! )

Deleing images from the paperclip attachments - Does indeed mark the image as removed from that one DB. but leaves the images on the server! They are only flagged as removed from that database but are left on the server using up your workspace UNLESS you also remove ALL Prior Suto and Manual backups relating to the current DB that also had links to those same images.

Ninox support have now confirmed - They way they are treating images handling needs revision and plan to release a fix at some point (there is no incentive to rush as their first gambit to me was - simply buy more space !!)

Removing images manually one at a time a pain in the backside! - Fortunately with @rosoft_steven help I now use a set of scripts that can sweep through one or more records to delete images in bulk within a given table .

So quick fix are :-

Remove the images tou don't want. The take min of 2 backups ! Then delete all backups Prior to those just taken that are foe=r thet DB (include any others you may c=have copied (ie i keep a copy of my LIVE db and call "play copy" but it links to same images (of Course).

Then depending on how you are deleted your images (ie by script) you may need one or more of the life line scripts Stephen helped me with! as these go on to absolutely flag them for deletion on the server (you still have to remove prior back ups until Ninox come up with a proper FIX!

SO auto removing images is either a one or two step affair

if all your images are on the paper clip it is a one step!

If held on image fields then it is two step affair

I use a dashboard (new pages!) to control removal

 

Some - Sample scripts (easily adaptable for your own use!)

Button - Deletes mutiple images in mutiple records in table called job dockets - This routine is to simply to remove the Field image files (NOT paperclip) from the jobs (soft delete them) in bulk. Another script will remove them (set the server to clear them)

THIS IS A TWO STEP PROCESS  - RUN THIS SCRIPT THEN RUN THE SCRIPT TO REMOVE PAPERCLIPS IMAGES AT SOME POINT LATER 

clears 6 image fields

let mySure := dialog("Sure?", "Are you Sure?", ["Yes", "No"]);
let myDate := today() - 150; lets me keep 150 days of records with the image still in

if mySure = "Yes" then
    for a in select JobDockets where Status = 4 or Status = 7 or Status = 6 and 'Order Date' < myDate do
        a.(SupplierQuote := null);
        a.(SupplierConfirmation := null);
        a.(SupplierVisual := null);
        a.(Image1 := null);
        a.(Image2 := null);
        a.(Image3 := null);
        a.(Image4 := null);
        alert("Images have been soft deleted from the system")
    end
else
    alert("Okay - Nothing has been deleted")
end

Button - To Delete images from Quotes File - Again soft deletes the image

Clear a single field image in bulk! (again not paperclip) and again a TWO step process

let myDate := today() - 150; lets me keep 150 days of records with the image still in
let mySure := dialog("Sure?", "Are you Sure?", ["Yes", "No"]);
if mySure = "Yes" then
    for a in select Quotes where Status = 5 or Status = 7 and QteDate < myDate do
        a.(QuoteImage := null);
        alert("Records have been deleted from the system")
    end
else
    alert("Okay - Nothing has been deleted")
end

 

Button - Delete soft deleted image files in Jobdockets ie get rid from the paperclip & the server - Remember these do not truly go if you have backups relating to the db that have same linked data !!

ONE STEP PROCESS 

USE WITH CARE !!!!

let mySure := dialog("Sure?", "Are you Sure?", ["Yes", "No"]);
if mySure = "Yes" then
    let method := "DELETE";
    let teamid := teamId();
    let dbid := databaseId();
    let tid := "B";
    let cont := "application/json";
    let bear := "Bearer " + first(select Settings).'API-Key';
    for i in select JobDockets where 'Order Date' <= today() - 400 do
        let rid := text(i);
        while cnt(files(i)) != 0 do 
            let file := urlEncode(text(i.Filename));
            let url := ---
https://api.ninox.com/v1/teams/{ teamid }/databases/{ dbid }/tables/{ tid }/records/{ rid }/files/{ file }
                ---;
            let response := do as server
                    http(method, url, {
                        Authorization: bear,
                        'Content-Type': cont
                    }, {})
                end;
            if response.error then
                alert(text(response.error))
            end
        end

    end
else
    alert("Okay - Nothing has been deleted")
end

I am very grate full and can't thank enough for the help from RoSoft_Steven.1 of which if he had not pointed out the initial issue and provided some of the excellent scripting -  I would still be dealing files one at a time .

Can't tell you how much this improved the admin of deleting file images from one of my other DB's that has over 800,000 records in it in one table !

So is it a bad thing that Ninox keep the files on their server (including any broken orphaned files! Well it depends on your point of view. If you deleted a load of images by mistake then reverting to a restored back up will put them all back.! But thereby hangs the issue the backups have that link to the images on the Ninox server thus You can delete as many images as you want but your workspace space will not be recovered UNLESS you delete all prior backups too.!

 

17 replies

null
    • Mel_Charles
    • 5 mths ago
    • Reported - view

    Delete PAPERCLIP ATTACHMENT in ONE record ONE Image at a time with each press of the button.

    let method := "DELETE";
    let rid := text(this);
    let teamid := DatabaseTeam;
    let dbid := DatabaseID;
    let tid := TableID;
    let file := urlEncode(text(Filename));
    let cont := "application/json";
    let url := ---
    https://api.ninox.com/v1/teams/{ teamid }/databases/{ dbid }/tables/{ tid }/records/{ rid }/files/{ file }
        ---;
    let bear := "Bearer " + first(select Settings).'API-Key';
    let response := do as server
            http(method, url, {
                Authorization: bear,
                'Content-Type': cont
            }, {})
        end;
    if response.error then
        alert(text(response.error))
    end

     

    Delete PAPERCLIP ATTACHMENTS - ALL Images with ONE press of the button.

    let method := "DELETE";
    let teamid := DatabaseTeam;
    let dbid := DatabaseID;
    let tid := TableID;
    let cont := "application/json";
    let bear := "Bearer " + first(select Settings).'API-Key';
    let rid := text(this);
    while cnt(files(this)) != 0 do 
        let file := urlEncode(text(Filename));
        let url := ---
    https://api.ninox.com/v1/teams/{ teamid }/databases/{ dbid }/tables/{ tid }/records/{ rid }/files/{ file }
            ---;
        let response := do as server
                http(method, url, {
                    Authorization: bear,
                    'Content-Type': cont
                }, {})
            end;
        if response.error then
            alert(text(response.error))
        end
    end

    • Haavard_Hjorteland
    • 5 mths ago
    • Reported - view

    Hi Mel, thanks for sharing, and thanks to Stephen for the script.

    I saw your reply in my <span onclick> thread some days ago. [BTW: the scrip  by rosoft_steven in that thread is working very well, it made a dream for me came true 👍)

    Before you posted this, I actually allready had deleted ALL my image fields and ALL my paperclips attachements. But I was stunned to see that by storage usage didn't reduse.

    So I am trying to do your  ONE STEP PROCESS  USE WITH CARE !!!!

    But there are a couple of things I dont understand.

    I have replaced your "B" for Table id with tableId(this), I believe this is not the problem.

    I  dont need a filter for what to delete, so I select all table records in my 'Personer' table with for i in select Personer do. Dont think that should be a probelem eighter.

    But I get the below error in:  in let bear := "Bearer " + first(select Settings).'API-Key'; << Do I need a API key? 

    also in :      let file := urlEncode(text(i.Filename));  << What is i.Filename refering to?

    • Mel_Charles
    • 5 mths ago
    • Reported - view

    File name is the name of the image file itself that has been stored on the server

    and yes you need a Ninox API key

    https://docs.ninox.com/en/api/introduction

     

    • Mel_Charles
    • 5 mths ago
    • Reported - view

    Haarvard I run mine from a dashboard (page).

    I have in fact removed the the filters and delete all paperclips (was too problematical).

    also with a very large table I have also noticed.

    when deleting images (not on the paperclip) they appear then on the paperclip.= but not in all cases seems to bee happing in 50% of the time.

    Running the above script then does indeed capture these spare created files for deletion.

    Again with my large table I find that it does not always update the table view and a log out and re log in is sometimes needed.

    and sometime the script itself drops out. Leaving some odds files left

    I use a check view to confirm files have gone.

    However with a test table of 200k records. I have re run the above more than twenty times throwing in all kinds of images both in the paperclip and other images files and can conclude that

    the sript to remove all images files (not paperclip) with a filter condition ie keep last 6 months etc. and the delete all paperclip attachement images is but a practicle and useful method of file handling to get rid of images.

    However in order to truly recover space Ninox are adamant that you also have to remove prev associated backups.

      • Haavard_Hjorteland
      • 4 mths ago
      • Reported - view

       Thank you for this.

      I have been in a state of dispare for the last three weeks - here is what happened, and I hope someone can guide me what to do next:

      My Genealogy DB Bygdebok_Kilder has for a long time uncontrolled increast in size , I'm pretty sure it is caused by the same as you describe. Pervios Image files/paperclips. I had been working to prepare your methode (removed ALL Image fields and deleted ALL paperclips)

      THEN: I did a stupid mistake, and deleted all 6000 records in my most importent Table with all the Persons. (this mistake was not relatded to this topic)

      So for three weeks I have been in email dialouge with Ninox support, because I could not recover my DB. They gave me temporary licensens, but that did not help, I could not recover the automated backup of my working DB. But HURRAY, some days ago Ninox developers  managed to restore it 👍 

      However, the DB size is still 5GB+The strange thing is: While I had temp 3-4 extra licenses I was able to do a manual backup of the "non working" DB , and then downloaded it. It was only 148 MEG in size!, even if it was a duplicate minus the 6000 records I deleted! 

      As shown in the screensot, this bacup have grown to 427MB without me created much new records , while my working Ninox-recovered DB is still 5GB+. 

      SO: Yesterday I asked if Support could give me temporary extra lisences again, so I could try to create a Manual backup - download it - delete the automatted backup - and upload it the day after.

      But here is supports reply: 

      "we understand that we have provided temporary licenses on several occasions in the past to assist with your requirements.  However, due to policy constraints, we regret to inform you that we are unable to provide additional temporary licenses at this time. In order to increase the storage space included in your subscription you can add more users to your subscription. Please see our prices on this page: https://ninox.com/en/pricing

      Thank you for your assistance."

      I refuse to pay for extra licenses for something I believe is a design flaw in Ninox regarding attachements/auto backups etc.

      Can someoone help me what to do now? My Genealogy DB is working again, but I can not do any backup that can be restored anymore 😧

      • Fred
      • 4 mths ago
      • Reported - view

      I don't know if this is solution, but if you have a MacOS computer, or a good friend that has one, the MacOS app, which is free, allows you to export entire Ninox DBs as is.

    • Haavard_Hjorteland
    • 4 mths ago
    • Reported - view

    Thank you,

    I dont have a MacOS computer, but could certanly borrow one an try this.

    I am on Publick cloud, can you import the Public Cload DB into the Ninox MAC app?

      • Fred
      • 4 mths ago
      • Reported - view

      Yes, you can access Public cloud in MacOS app. You just login to the app as you would the cloud. That is what I do.

      I would love to continue to use the app but it is stuck on 3.9. The mobile apps are even further behind. As they made the app free and stop making the app match the cloud, I transitioned to the cloud. Now they are focusing on Private vs Public so we are back to not having matching features. No way I can afford a Private license.

    • Mel_Charles
    • 4 mths ago
    • Reported - view

     Sorry to hear that you having so many issues. re deleting records accidentally - we have all done this - We have all been there. I once developed one of my DB's then decided I did not need all the backups and an hour lated I inadvertently deleted the DB in error ! whooops !

    also when testing  an email send script one day _ I asked a customer if i could send her a few test mails from the system vis sendEmail - she agreed. so I hard wired her email into my script. It worded - The next day I had forgot this and sent her over 10,000 emails !!!! 

    I can't really add anything that Fred has not already said but could the missing 6000 records have been holding lots of images and hence the sizing difference?

    Also as Ninox offer a 30 day trial account on cloud. could you not register and other email and the upload the restore yur DB to that new site and sort? I guess in as a worse case then you could also save an extra backup an emergency copy by unticking backup without attachments to give you a data only version as extra backup which would then be very small in terms of sizing. which is on option on the cloud version

      • Haavard_Hjorteland
      • 4 mths ago
      • Reported - view

       I have borrowed a MacBook Pro, installed the Ninox app, and logged in succesfully. The app is functioning OK.

      1: I still get the "Storage full" alert when trying to do a "Crate Backup" directly from the workspace (as excpected)

      2: Next I tried to select the DB, and then in the top menu selcect "Database > Save archive as", and point it to the Mac download folder. But i get Ninox error message "NOT IMPLEMENTED" 

      3: Third option I have tried is: Selected the DB, then "Export data", then selected "NINOX" and tried different options with 'Data model', 'Data' and 'Files' checkboxes. + Save as to MAC download folder. But when trying this, the "spinning wheel" goes forever, and the app seems to chrash.

      I have also tried this with a very small test DB, but it still is only spinning wheel  and nothing will download.

      Do you have any suggestions?

      thanks and regards,

      Håvard

    • Fred
    • 4 mths ago
    • Reported - view
     said:
    3: Third option I have tried is: Selected the DB, then "Export data", then selected "NINOX" and tried different options with 'Data model', 'Data' and 'Files' checkboxes. + Save as to MAC download folder. But when trying this, the "spinning wheel" goes forever, and the app seems to chrash.

     This is the method that works for me.

    I'm not sure why it is not working. When I check my Mac, I don't give Ninox permission to Files or Folders or Full Disk Access so I don't know why it is allowed to write. You may have to write to Support.

      • Fred
      • 4 mths ago
      • Reported - view

        Well it looks like you are using 3.10.10 of the MacOS app and I was on the old 3.9.x. I just upgraded to 3.10.10 and I too get this problem now. 😫

      I sure hope they fix this soon.

    • Mel_Charles
    • 4 mths ago
    • Reported - view

    Haavard Hjorteland Apologies, I missed the fact that you were talking about the App version. I'm totally Web.

    I have no idea how the app/iStore version works in terms of archived back up's - Do you not have any backups at all that you can roll back to even if say a month old that was prior to the last bloat in size. as presumably all was working back. I rather have that wit say a loss of a few weeks and to rekey x records than have the loss 6000 records. 

    I guess at the end of the day as your working space is effectively zero - you can't perform operations like backups etc - If it were me and I appreciate it is not and I don't know how critical the data is to you, whereas my data is for running my business - thus vital to me then I would pay for the extra license get to get the space headroom in order to to be able to get my data back. Cos if you have to start again on that many records. The ticketed labour and time lost in percentage terms is potentially high.

    Anyone?- Does Ninox support not actually see the backups on the app form a server point of view. 

    I'm probably talking gibberish here, but given you said you have a file of 5GB (full db) and. File of 148 meg (minus 6k record)

    As you now say you got in okay - Do you just have the option to just delete the Large 5GB version? Then you have space to rerun and back up the lessor version -  I know you will have 6k records short but at least that would be better nothing or restore an earlier back up prior to the 3 weeks/6k record loss ie before you tried to delete the images etc.

    I do feel for you my friend!

      • Haavard_Hjorteland
      • 4 mths ago
      • Reported - view

      I have just resolved my problem ! 👍

      I am on Public cloud Pro. Ninox support once again refused to give me Temporary licenses, but insured me that I can reduce to one license again next year. So i had to pay for two extra , witch cost me a great deal of money.  But I just had to do it this way, as I have not been able to do manual backups for a LONG time. 

      Once I got 3 licenses I was able to Manual backup my working but bloated DB, and download it. Guess what? Size was just 134Kb! For now I have only imported it back in a Free trial account i created, and it works as a charm. I will delete the bloated DB tonight, and remove all Auto backups and see tomorrow.

      I told Ninox support several times that the DB size probably was caused by previous used image files and auto backups, but the answer was always  "Upgrade" or "delete some data". I could not find any data to delete...  So I have a slight feeling of "some wasted money", since this seems to be of Ninox's design.

      My lessons learned is: I will do at least weekly manual backups and download, and not only  rely on the automated backups.  

      I have one question: When I first started using Ninox I was foolish enough to upload a large Image under Table>Options>Background Image>Upload Image. I have this image also in a 10Meg +/- size version from a camera. I hope I uploaded a smaller version, but I am not sure, and don't know how to check.

      Is there a way to get rid of this background Image? If i just reupload a small tiny Image I'm afraid the old one is still added to my auto backups?

    • Fred
    • 4 mths ago
    • Reported - view
     said:
    Is there a way to get rid of this background Image? If i just reupload a small tiny Image I'm afraid the old one is still added to my auto backups?

    I can't answer the first part as I don't have a pro license.

    You are correct in the second part. If you upload the DB with the large image then allow an auto backup to happen, then as long as you have a backup of the version with the image it will keep it in the backup. Just in case you wanted to return to that version.

    Seems like you can upload a fresh DB, delete/replace the image before a backup happens then you should be OK.

      • Haavard_Hjorteland
      • 4 mths ago
      • Reported - view

       Thanks,

      It was as excpected, but I could not find a way to delete this background image. However, when i Create a Manual backup and uncheck "Include files" it seems to be removed. So I'm Okay.

      In the future I will be very careful and try to avoid Image Fields and file attachements. In fact I have zero right now, I use other methods for my images gallery, maps etc.  I understand that it is meaningful to keep images/attachements in automated bacups, but Ninox should have implemented a way to permantly delete those. Eighter one by one, or a list with checkboxes and  "permanently Delete all selected files? " or something like that. 

      Since I couldn't figure out how to make Mel and RoSoft_Steven's workaround above work,  I added this to my Persons table to ease the task of deleting paperclips:

      1: A FX called 'cntfiles' with : count(files(this))

      2: A View element with: select Personer where cntfiles > 0

      Then I could popup records that had paperclip attachement(s) , and delete it.

      But as Mel stated above, it is only a soft delete. At least i is a way to keep track of witch records has attachements.

    • Mel_Charles
    • 4 mths ago
    • Reported - view

    Thats great news - I did think that your only solution would be to pay for the upgrade to give you more GB headroom - So I hope you actually got back the the DB prior to the loss of the 6k records.

    You can definitely drop back down the size next year. I have increased and decreased my size as I have needed.

    When Ninox say delete some data - they don't mean spurious orphaned data floating around on the server. They mean within your DB and or backups etc

    The trade offs are:-

    re the images - you delete them in a current db but whilst you have backup(s) in place that refer to those same images, they still sit on the server, after all if you restore and all your images did not come back you would be pretty miffed. Hence when I said purge them from the current db. Do this then BACK UP (TWICE!). Then remove all previous backups that related to the original bloated file.

    Also from a data point of view. If you say have a 2GB limited and you are getting near the size limit and don't want to pay for more space then your options are. Delete images/delete reraly records/ delete backups

    My golden rules are:

    Before I run any script that each batch import updates many records/Modify files many records (ie change status on all records that have been booked out to archive records or batch delete records/ or images - BACKUP!

    Only keep max 6 auto backups at anyone time - 

    Do 2 manual backups before I make any significant structural changes to an existing table. If after work is done and okay. then I delete the earlier on leaving 1 in place.

    Once a week I download a backup to my desk top. No Fail

    Anyway - very pleased you are back in the land of the living ! 👍

Content aside

  • 4 mths agoLast active
  • 17Replies
  • 257Views
  • 3 Following