everytime i check nginx logs its more scrapers then i can count and i could not find any good open source solutions

  • daniskarma@lemmy.dbzer0.com
    link
    fedilink
    English
    arrow-up
    4
    ·
    edit-2
    9 hours ago

    How do you know it’s “AI” scrappers?

    I’ve have my server up before AI was a thing.

    It’s totally normal to get thousands of bot hits and to get scraped.

    I use crowdsec to mitigate it. But you will always get bot hits.

    • Drunk & Root@sh.itjust.worksOP
      link
      fedilink
      English
      arrow-up
      1
      ·
      4 hours ago

      bot hits i dont care my issue is when i see the same ip querying every file on 3 resource intensive sites millions of times

      • daniskarma@lemmy.dbzer0.com
        link
        fedilink
        English
        arrow-up
        1
        ·
        edit-2
        3 hours ago

        Do you have a proper robots.txt file?

        Do they do weird things like invalid url, invalid post tries? Weird user agents?

        Millions of times by the same ip sound much more like vulnerability proving than crawler.

        If that’s the case fail to ban or crowdsec. Should be easy to set up a rule to ban an inhumane number of hits per second on certain resources.

  • some_guy@lemmy.sdf.org
    link
    fedilink
    English
    arrow-up
    1
    ·
    7 hours ago

    I don’t have opensource solutions, but CloudFlare had some news about a system that I didn’t read about (saw two headlines) last week. Dunno if it works or not.

  • kcweller@feddit.nl
    link
    fedilink
    English
    arrow-up
    12
    ·
    13 hours ago

    Nephentes that shit. Poison every scraper until they start respecting robot.txt. Purposefully use llm.txt to trap the fuckers.

      • groet@feddit.org
        link
        fedilink
        English
        arrow-up
        1
        ·
        6 hours ago

        You most likely pay for a maximum CPU capacity and your Server cant go above that no matter what you run. Your vps provider doesn’t care what that CPU power is used for.

        However with limited resources, the tarpit might use up all CPU power and the rest of the webserver will crawl to a halt.

    • Drunk & Root@sh.itjust.worksOP
      link
      fedilink
      English
      arrow-up
      3
      ·
      20 hours ago

      i can only get it to protect one container. i have 3 that i need protected and i cant figure out how to run more then one instance of it.

  • gandalf_der_12te@discuss.tchncs.de
    link
    fedilink
    English
    arrow-up
    2
    ·
    edit-2
    20 hours ago

    What’s bothering you?

    • Is it to give out data for AI training? I guess you can’t fundamentally protect against this, except by limiting how much content is provided to each address.
    • Or is it the resource strain that it causes on your server? In that case i recommend limiting how much a single client / IP address can request in a day.
  • Fedditor385@lemmy.world
    link
    fedilink
    English
    arrow-up
    14
    ·
    2 days ago

    Anubis is the name of the tool. Also, Cloudflare just announced they have something against AI scrapers.

    • Drunk & Root@sh.itjust.worksOP
      link
      fedilink
      English
      arrow-up
      2
      ·
      20 hours ago

      ive been using Anubis my only issue is i would have to run more then one instance and i dont like cloudflare personaly

  • Fedditor385@lemmy.world
    link
    fedilink
    English
    arrow-up
    7
    ·
    1 day ago

    I just realized an interesting thing - if I use Gemini, and tell it to do deep research, it actually goes to the websites it knows/finds, and looks up the content to provide up-to-date answers. So, some of those AI crawlers are actually not crawlers, but actual users who just use AI instead of coming directly to the site.

    Soo… blocking AI completely could also potentially reduce exposure, especially as more and more people use AI to basically do searches instead of browsing themselves. That would also explain the amount of requests daily - could be simply different users using AI to research for some topic.

    Point is, you should evaluate if the AI requests are just proxies of real users, and blocking AI blocks real users from knowing your site exists.

    • daddycool@lemmy.world
      link
      fedilink
      English
      arrow-up
      14
      ·
      1 day ago

      some of those AI crawlers are actually not crawlers, but actual users who just use AI instead of coming directly to the site. Soo… blocking AI completely could also potentially reduce exposure.

      Normally, websites want users to come to their site, instead of an AI search engine “stealing” the content and presenting it as it’s own. Yes, AI search engines are more convenient for the user, but in the end it will discourage website creators and thereby cut of it’s own “food supply”.

      • Zexks@lemmy.world
        link
        fedilink
        English
        arrow-up
        1
        ·
        8 hours ago

        We all understand that. But if those users keep insisting on giving everyone their life story and current option in world politics before giving us the bread recipe we came for, they can fade away.

      • nfreak@lemmy.ml
        link
        fedilink
        English
        arrow-up
        13
        ·
        1 day ago

        Yeah I’d consider blocking out both the bots and AI-users a win-win lmao

      • Fedditor385@lemmy.world
        link
        fedilink
        English
        arrow-up
        3
        ·
        1 day ago

        I understand, but the shift in user behaviour is significant and I think websites are not taking it into account. If the users move more and more to AI, and since Google introduced AI mode it’s only a question of time until it becomes the default, we will see more and more of what we thing are AI crawlers and less and less organic users.

        AI seems to be the new middleman between you and the user, and if you block the middleman, you block the user. For people with hobby websites or established sites it may make sense because people either know of them, or getting more exposure is not a wish or requirement, but for everyone else, it will be painful.

        • lambalicious@lemmy.sdf.org
          link
          fedilink
          English
          arrow-up
          2
          ·
          21 hours ago

          So, what I’m reading is, if your “users” are bad (or bots), just get better users.

          Sounds like a net win.

        • Noja@sopuli.xyz
          link
          fedilink
          English
          arrow-up
          2
          ·
          1 day ago

          I honestly don’t think most people replace search with AI, it will also slowly solve itself when google injects ads into the output.

    • rumba@lemmy.zip
      link
      fedilink
      English
      arrow-up
      5
      ·
      1 day ago

      Porque no los dos?

      There is no functional difference between them scraping you systematically and them coming to you on behalf of user. They’re coming to scrape you either way, being asked by someone is just going to make them do it in a smarter fashion.

      Also, if you’re not using Gemini, damned if Google.com doesn’t search you with it anyway. They want these AIs trained bad, sooner or later almost all searching will be done through AI. There will eventually be no option.

      You are correct that blocking all AI calls well eventually make your search results not work.

      So if you want organic traffic, you have to allow ai scraping eventually. You’re just going to get diminishing returns until a point.

      • youmaynotknow@lemmy.ml
        link
        fedilink
        English
        arrow-up
        3
        ·
        1 day ago

        Eso es correctísimo. I don’t want ANY AI in my servers looking for anything, regardless of if they are crawlers or if it’s on behalf of some lazy fuck.

    • Mordikan@kbin.earth
      link
      fedilink
      arrow-up
      12
      ·
      2 days ago

      I especially love the irony of Anubis using yesterday’s hype thing to combat today’s.

    • RedBauble@sh.itjust.works
      link
      fedilink
      English
      arrow-up
      4
      ·
      edit-2
      2 days ago

      Second Anubis, just finished by setup yesterday i have it of a oracle cloud frre tier vps, which depending on the domain routes the traffic to services hosted on the vps itself or to my server ar home. Relatively easy to setup, blocks most requests with very few false positives (one of which for example it would aggressively challenge by thunderbird trying to reach my baikal instance). I set a bit more aggresive rules than default (i also block googlebot and bingbot, since i received a bit more requests than I’d like). In like 10 hours it straight up denied about 5000 requests from the ai-catchall ruleset (mostly amazonbot) and challenged about 10000, mostly from a block of IPs in singapore, some of the hosts having the user agent of a Macintosh with PowerPC. They all sure love to explore the public repos on my git server.

      I’m in the process of changing servers for an upgrade, the old one still hosting more services while I setup the new one. The old one now does run audibly quiter. I don’t even want to think how much electricity went wasted because of those bots

      • VeganCheesecake@lemmy.blahaj.zone
        link
        fedilink
        English
        arrow-up
        5
        ·
        1 day ago

        You probably don’t need me to tell you, but keep good backups. Friend of mine recently had his account nuked without any reason given, and without the possibility of recourse.

        a mail from Oracle, informing about the immediate termination of service, and deletion of all data

        • RedBauble@sh.itjust.works
          link
          fedilink
          English
          arrow-up
          1
          ·
          edit-2
          14 hours ago

          That’s too bad. Luckily i keep just a couple of docker compose stacks there. But I should start backing them up, that vps is the only thing I don’t backup

  • SayCyberOnceMore@feddit.uk
    link
    fedilink
    English
    arrow-up
    6
    ·
    2 days ago

    If you’re able to, use GeoIP ranges to only allow access from the countries you want.

    That immediately limits a lot of everything

    Then - again if you’re able to - use a block list that covers known scrapers in case they’re in your country.

    I use pfBlockerNG on my pfSense firewall for exactly this.

  • db0@lemmy.dbzer0.com
    link
    fedilink
    English
    arrow-up
    20
    ·
    2 days ago

    You need yo block the alibaba subnets primarily. In my experience this is where most of them originate

  • Igilq@szmer.info
    link
    fedilink
    English
    arrow-up
    3
    ·
    1 day ago

    Well, someone had great idea to use zipbombs. I saw it somewhere but I don’t remember where.

  • Bahnd Rollard@lemmy.world
    link
    fedilink
    English
    arrow-up
    14
    ·
    2 days ago

    Wern’t there a few AI maze projects in the works? I wonder if running one of those for a bit will cause you to be added to an ignore list, clearly they dont respect your robots file.