# System Administration You get to mess with computers and make money. I think that's pretty cool [Task management](Task management) [Time management](Time management) [IaC](IaC) # Small batches TODO preliminary notes, include more examples and details Work in small batches Iterate quicker Easier to find problems/bugs in a smaller amount of work Can pivot easily if need arrises Increases motivation due to visible progress Document risky processes, try to automate them, what makes them risky? Perform scheduled downtime to practice the process and try to improve it MVP strategy: least amount of changes to make it viable # Software suites Maybe one day you'll need to deploy some kind of software suite that encompasses your general office activities: word docs, presentations, spreadsheets and other kind of software. Most of the time these need to support collaboration besides simply sharing the files, some kind of live view/edit/comment workflow. In that case, some useful FOSS software of the kind: * https://cryptpad.org/ -> France-based, security first E2EE general office suite # Control panels [HestiaCP](https://demo.hestiacp.com:8083/list/user/) is a relatively lightweight (min 1GB RAM) control panel with a cool interface. # Planning Planning anything, even if it seems to be the simplest thing, is a must. Use checklists (see https://utcc.utoronto.ca/~cks/space/blog/sysadmin/UseAChecklist) to ensure you have an in-depth understanding of everything there is to come, so when you get caught off guard (not if, but when) you will be prepared (if what you wrote down is accurate!) Remember, there will always be surprires. See also https://utcc.utoronto.ca/~cks/space/blog/sysadmin/SysadminAphorism # Diskless clients Diskless clients are regular PCs without means for permanent storage. They usually boot from a central gateway that hosts something like a network boot loader (iPXE), DNS, DHCP, NFS and other services. LTSP at https://ltsp.org/ can be useful for deploying linux diskless nodes. # Microsoft... Free MS Dev stuff: - 1TB onedrive storage (slow) - Office 365 25 licenses - Azure devops - unlimited private git repos # Email server See [email](email) # LLM/Crawler protection TODO add some kind of verification at 404? So normal people aren't affected by this, if they happen to visit a nonexistant page, give them a chance before kicking them out - Implement something like arch wiki account creation, a question that could change once in a while that requires running a specific command - Maybe just steal it from the arch wiki if it's openly licensed (should be) Mostly [fail2ban](fail2ban), see also projects such as https://zadzmo.org/code/nepenthes/ See locaine at https://iocaine.madhouse-project.org/ that serves garbage to bots while being very efficient. The ethical choice to deter bad actors. Seems like locaine does a good job of filtering out real useragents from fake ass slop, inexistent, unrealistic user agents. See vulpinecitrus' article on defenses for more info at https://vulpinecitrus.info/blog/guarding-git-forge-ai-scrapers/ Also includes a tidbit about caching content in nginx, albeit with some downfalls related to successful service auth. See Alex Schroeder's Butlerian Jihad guide on effective blocking by ASN: https://transjovian.org/view/fight-bots/index * What if I just ban all ASNs except Romania for personal sites, alongside with more protection? (permanent instakill on 404|418 with fail2ban) * https://lobste.rs/c/3vovci - Serve untrusted users pages without hyperlinks - Login to see full content ## HTTP basic auth Experimented with requiring HTTP basic auth for forgejo on [sundown](sundown). It does stop crawlers running expensive git operations (since they can't access the forge at all without the user,pass pair) however it has some disadvantages. It requires people to either: - input user and pass on each browsing session - have to save credentials somewhere, using builtin browser passmanager (vendor lockin) or external password managers (slowness) - browsers do not automatically submit auth form (slowness) - addons exist, but it's stupid to install addons just to visit a git forge (inconvenient) - use a special link of the form `https://user:pass@git.ctq.ro` - bookmark for easy visit - was promising, however source map errors regarding embedded credentials are problematic (bugs) - can workaround by clicking on url and reloading the page manually (removing the embedded credentials) (inconvenient) ## Use javascript to get a cookie A simple stopgap measure: make users get a cookie by running some javscript + transparent + simple + no additional daemons - requires javascript - requires cookies As seen here: https://yannesposito.com/posts/0031-how-i-protect-my-forgejo-instance-from-ai-web-crawlers/index.html ## go-away middleware between website content and proxy (like iocaine) modify example forgejo policy, remove some problematic challenges like meta-refresh TODO this does not work reliably as of 2026-04-07 with forgejo, even with most challenges removed it still fails ### no-js can use only non-javascript challenges As seen here: https://jade.ellis.link/blog/2025/05/18/actually-stopping-forgejo-ai-scraping # Git gitolite at https://gitolite.com makes it easy to self-host git. Just install and configure according to docs. Wild repos are a useful feature so you can quickly create remote repos locally without SSH-ing in. ```sh @creators = archie repo CREATOR/[a-z]..* C = @creators RW+ = CREATOR RW = WRITERS R = READERS ``` To create a new repo, you can run `git push -u git@SERVER:USER/REPO main` See also -------- * https://www.opsschool.org/index.html -> sysadmin curriculum * https://github.com/awesome-foss/awesome-sysadmin -> useful software https://rys.rs/books/wiki/page/ -> some online notes