I'm working on an old Laravel project with a reasonably large data set (22Gb+) that cannot be replicated on a testing server with with migrations and seeds with faker factories.
There are daily mysqldumps performed early in the morning and stored on an s3 bucket.
I don't want to setup an extra mysqldump that obfuscates personal data and then import that to my test server as this would cause too much load on the production server. What I want is to use the test server to copy and manipulate that mysqldump and obfuscate any personal data then import it.
Options:
- I could load the entire mysqldump into a DB and process the data. I don't like this as there would still be personal data being loaded into the DB as well as this being an expensive operation but might be my only option at this point.
- Read stream / load file in a class then find and replace the data sets, I don't like this as loading in a 20gb+ file (even line by line) might make PHP run out of RAM and kill itself
- ProxySQL has been suggested elsewhere but is well known not to play well with laravel
