Inkbunny launches South African cache

The place to talk about anything just generally furry. If you have a furry-related thread that doesn't fit anywhere else, put it here! Just keep it clean.
Post Reply
User avatar
GreenReaper
Posts: 23
Joined: Sun Nov 30, 2008 10:01 am
Gender: Male
Sexual preference: bi-mostly straight
Species: Kai Norn (Creatures)
Region: Other
Location: London, UK
Contact:

Inkbunny launches South African cache

#1

Post by GreenReaper »

Hi all! Over the past few months, furry art community Inkbunny (wiki) has been on a quest to improve performance, deploying public file caches in the USA and Australia, as well as Canada, Japan and Brazil. Today we launched one in Johannesburg! :-)

It's not super-powerful - a 50%-core Xeon E5-2430 with 512MB RAM and 30GB on a HDD w/SSD cache - but it can serve up to 500GB/month at 50MBps from 10-40ms away, rather than 180ms from the Netherlands.

The cache serves South Africa, Zimbabwe, Botswana and Maputo. One Durban member saw download speeds improve 50%+ on top of an ~80% cut in access times. Of course, the more furs using it, the fresher it'll be, so we're hoping you'll all drop by to give it a spin.

We weren't originally planning to expand this far, but we get over 1500 monthly visits from South Africa, and we couldn't pass a 50%-off coupon from Host1Plus. (They're still offering it via code VPSZA50 for terms up to a year, if anyone needs a similar server.)

If you have an FA gallery, you might also like the FA to IB migration tool - it's run by our staff, and faster and cheaper than uploading and describing everything again yourself. Just check our content policy before publishing, as it differs in some respects.

Hope to see some of you on Inkbunny soon! :-)

P.S. Anyone using a bulk downloader may want to set the Netherlands cache in the settings of the account they use for it, as this new cache can only store a limited number of original-sized files - it'd be faster to get them from the source.
Laurence "GreenReaper" Parry
GreenReaper Studios - WikiFur - Flayrah - Inkbunny
User avatar
Valerion
Alpha Wolf
Posts: 2803
Joined: Fri Apr 11, 2008 8:50 pm
Gender: Male
Sexual preference: Gay
Species: Werewolf
Region: Gauteng
Location: ::1
Contact:

Re: Inkbunny launches South African cache

#2

Post by Valerion »

Very nice!
User avatar
Rakuen Growlithe
Fire Puppy
Posts: 6427
Joined: Tue Apr 01, 2008 2:24 pm
Gender: Male
Sexual preference: Bi
Species: Growlithe (pokemon)
Region: Gauteng
Location: Pretoria
Contact:

Re: Inkbunny launches South African cache

#3

Post by Rakuen Growlithe »

I didn't expect SA was a big enough market for this. Well done, guys and thanks Greenreaper. :)

PS: I bought a Inkbunny sweatshirt.
User avatar
Sev
Superbike Snow Leopard
Posts: 6596
Joined: Thu Mar 06, 2014 9:27 pm
Gender: Male
Sexual preference: Gay
Species: Snow Leopard
Region: Western Cape
Location: A Twisty Road

Re: Inkbunny launches South African cache

#4

Post by Sev »

You guys just earned more visits from me. :D
User avatar
Sticky_Fingers
Posts: 360
Joined: Sat Sep 20, 2008 12:30 am
Gender: Male
Sexual preference: Bi-mostly gay
Species: brown bear
Region: Western Cape

Re: Inkbunny launches South African cache

#5

Post by Sticky_Fingers »

This is absolutely amazing. I will probably try to get more of a presence on inkbunny then if it will help.
Just your average internet denizen with a strange name and dreams of becoming a game developer. Catch me on FA: stickyfingers
User avatar
GreenReaper
Posts: 23
Joined: Sun Nov 30, 2008 10:01 am
Gender: Male
Sexual preference: bi-mostly straight
Species: Kai Norn (Creatures)
Region: Other
Location: London, UK
Contact:

Re: Inkbunny launches South African cache

#6

Post by GreenReaper »

Sticky_Fingers wrote:This is absolutely amazing. I will probably try to get more of a presence on inkbunny then if it will help.
We have plenty of room for greater use in South Africa, and submissions; both are beneficial to others! :-)

I don't know how you'd feel about sharing some or all of your older art with our members, using FA to IB or manually; regardless, you might find a few interesting artists to watch.
Laurence "GreenReaper" Parry
GreenReaper Studios - WikiFur - Flayrah - Inkbunny
User avatar
GreenReaper
Posts: 23
Joined: Sun Nov 30, 2008 10:01 am
Gender: Male
Sexual preference: bi-mostly straight
Species: Kai Norn (Creatures)
Region: Other
Location: London, UK
Contact:

Re: Inkbunny launches South African cache

#7

Post by GreenReaper »

Good news! While our prior cache lapsed in mid-2016 after our coupon's expiry - IB has a global budget of ~$10/day, which only goes so far, and it was hard to get a VM that wasn't more than we needed in the area - we've had our eye on offers from cloud providers since then, and last year it became possible to use one of them in Johannesburg1 (as of now, there's more than one in South Africa). Once our server move was out of the way, we decided to have a closer look.

With South Afrifur's recent cancellation and ongoing lockdowns, this seemed the perfect time to deploy a replacement. There's been however a slight delay caused by negotiating our host's own lockdown - they're currently restricting VM deployment to select customers due to capacity concerns. We've noted South Afrifur's cancellation as justification for our use of the region, and hope that'll be adequate; until we get a decision, we're using our credit with them to use their CDN instead, which has nodes in Cape Town, Johannesburg, Lagos and Nairobi. This will probably just be for a few days until it's up.

Five years on, the specification has been upgraded to 68GB of full low-latency SSD, 1GB RAM and a burstable CPU. We suspect regional network connections have also improved in recent years, hopefully we can use this content server for additional states in southern, central and/or eastern Africa (e.g. Kenya, Nigeria), but it'll depend on the results of testing when we actually have the server in hand.

Due to limited storage and transfer, we may end up serving only thumbnails, website layout and previews from Joburg; the rest coming from Europe as usual. It'll depend on how much traffic is used by it. Outbound transfer in South Africa costs 150-200% that of most regions, equal only to Brazil and the UAE; for non-cloud providers who charge closer to the underlying rate, it's more like 1000%. 😾

(Curiously, serving from the VM via CDN costs 45% of what it'd cost to serve direct from the VM, because a) for CDN traffic, Africa's part of EMEA, not a separate zone; and b) VM -> CDN traffic is inclusive - we hope to use this to our advantage to e.g. serve website backgrounds via the CDN, saving our normal monthly allowance for thumbnails etc. Or we may just keep serving them from Europe.)

I appreciate an online community can't replace the experience of meeting friends and fans in person - and selling them art, comics, textiles and trinkets - but I know a few of you use IB; hopefully it helps! We'll try to keep this up at least until the next convention approaches in 2021; after that point, we'll see what offers are available to us.

1 Ironically one of the datacenters our new host uses is just to the south of Croydon (near Islando) in Ekurhuleni; I went to school in South Croydon, albeit in the UK!
Last edited by GreenReaper on Thu Apr 30, 2020 5:46 pm, edited 1 time in total.
Laurence "GreenReaper" Parry
GreenReaper Studios - WikiFur - Flayrah - Inkbunny
User avatar
Rakuen Growlithe
Fire Puppy
Posts: 6427
Joined: Tue Apr 01, 2008 2:24 pm
Gender: Male
Sexual preference: Bi
Species: Growlithe (pokemon)
Region: Gauteng
Location: Pretoria
Contact:

Re: Inkbunny launches South African cache

#8

Post by Rakuen Growlithe »

I don't really understand all of that but yay! Getting our own cache is awesome. Again!

Actually Inkbunny has always been really fast. It wasn't a few weeks back but that was when there was a break in one of the major undersea cables connecting South Africa to the rest of the world. Some sites weren't too badly affected but Inkbunny become unbearably slow. Luckily that's all fixed now and I guess Inkbunny will be even faster! :)
User avatar
GreenReaper
Posts: 23
Joined: Sun Nov 30, 2008 10:01 am
Gender: Male
Sexual preference: bi-mostly straight
Species: Kai Norn (Creatures)
Region: Other
Location: London, UK
Contact:

Re: Inkbunny launches South African cache

#9

Post by GreenReaper »

Rakuen Growlithe wrote:
Thu Apr 30, 2020 5:44 pm
Actually Inkbunny has always been really fast. It wasn't a few weeks back but that was when there was a break in one of the major undersea cables connecting South Africa to the rest of the world. Some sites weren't too badly affected but Inkbunny become unbearably slow.
Good to know! Inkbunny's main server - and the cache you'd probably have been using previously - is in Europe; I suspect most other furry services are based in the USA. That'd normally be to our advantage, as South Africa is "only" 150-180ms from Europe; but if the break meant they had to route via Singapore and the USA to get back there, it could easily have tripled that. We were also dealing with a DDoS for most of March, which probably didn't help, especially if your Internet connection doesn't support IPv6 (or it does, but the route was congested).
Laurence "GreenReaper" Parry
GreenReaper Studios - WikiFur - Flayrah - Inkbunny
User avatar
Rakuen Growlithe
Fire Puppy
Posts: 6427
Joined: Tue Apr 01, 2008 2:24 pm
Gender: Male
Sexual preference: Bi
Species: Growlithe (pokemon)
Region: Gauteng
Location: Pretoria
Contact:

Re: Inkbunny launches South African cache

#10

Post by Rakuen Growlithe »

GreenReaper wrote:
Thu Apr 30, 2020 5:55 pm
Good to know! Inkbunny's main server - and the cache you'd probably have been using previously - is in Europe; I suspect most other furry services are based in the USA. That'd normally be to our advantage, as South Africa is "only" 150-180ms from Europe; but if the break meant they had to route via Singapore and the USA to get back there, it could easily have tripled that. We were also dealing with a DDoS for most of March, which probably didn't help, especially if your Internet connection doesn't support IPv6 (or it does, but the route was congested).
I might have missed something but I think I looked into it and my ISP doesn't seem to have IPv6 at all. I don't think many in South Africa offer it either.
User avatar
GreenReaper
Posts: 23
Joined: Sun Nov 30, 2008 10:01 am
Gender: Male
Sexual preference: bi-mostly straight
Species: Kai Norn (Creatures)
Region: Other
Location: London, UK
Contact:

Re: Inkbunny launches South African cache

#11

Post by GreenReaper »

Rakuen Growlithe wrote:
Thu Apr 30, 2020 5:58 pm
I might have missed something but I think I looked into it and my ISP doesn't seem to have IPv6 at all. I don't think many in South Africa offer it either.
No, that's probably right; there's almost zero IPv6 penetration in South Africa (or, for that matter, the rest of Africa). I suspect the reason for this is that IPv4 addresses have been sufficient, as Africa was allocates its own blocks of addresses - mobile users are likely to share them via a NAT provided by their telecoms provider.

Our host's CDN has IPv6, but the VM does not by default; nor for that matter does our one in Seoul. IPv6 penetration is similarly low there, also there's no great reason to do so. Eventually this will change as shortages become more acute.
Laurence "GreenReaper" Parry
GreenReaper Studios - WikiFur - Flayrah - Inkbunny
User avatar
GreenReaper
Posts: 23
Joined: Sun Nov 30, 2008 10:01 am
Gender: Male
Sexual preference: bi-mostly straight
Species: Kai Norn (Creatures)
Region: Other
Location: London, UK
Contact:

Re: Inkbunny launches South African cache

#12

Post by GreenReaper »

Looks like it's working! For the curious, here's the hit/miss data from the last hour highlighting the CDN point-of-presence used to serve each request.
(JNB == Johannesburg, CPT == Cape Town, LON == London, AMS == Amsterdam)
LogQuery.png
LogTable.png
LogTable.png (4.01 KiB) Viewed 595 times
Hit rate isn't so good if you measure by bytes, but that's expected; a lot of content will be a miss, especially in the first few hours/days.

More odd is the presence of London and Amsterdam (I've seen Manila and Hong Kong there, too).
Doesn't matter for now since it's all coming from the Netherlands Cache until our VPS is up, but I'll have to see if I can redirect those to different origins closer to the PoP.
What we don't want is people going to the Netherlands (CDN), which fetches from South Africa (cache server VPS), which in pulls it from Europe.
Laurence "GreenReaper" Parry
GreenReaper Studios - WikiFur - Flayrah - Inkbunny
User avatar
GreenReaper
Posts: 23
Joined: Sun Nov 30, 2008 10:01 am
Gender: Male
Sexual preference: bi-mostly straight
Species: Kai Norn (Creatures)
Region: Other
Location: London, UK
Contact:

Re: Inkbunny launches South African cache

#13

Post by GreenReaper »

The storage/CDN VM inside South Africa was activated yesterday; requests not found in the CDN go to the VM in Johannesburg! Of course the first time anyone makes a request, it always has to pull the data from Europe anyway, but the idea is it only has to do that bit once, and then it just serves it from Johannesburg every time.

However, I found an issue with how the CDN nodes in South Africa access the VM - basically, it was going all the way to London to check the CDN node there, if it's not already in the local node! You can see this in the above graphs - turns out, almost all those London requests were cache misses from Johannesburg and Cape Town. Then, since it's probably not in London, it goes back to the Johannesburg VM... and if it's not there, it goes back to Europe to the main servers... 😾)

I implemented a workaround for this, to skip even trying to load from or save to the CDN cache, and route directly from the CDN nodes to the VM. Thumbnails and usericons you haven't seen before should load as well as possible now - though they'll all load again, once, because of the change I had to make; and if you're the first in South Africa to request a particular thumbnail/icon, you'll still have to wait a little bit.
Laurence "GreenReaper" Parry
GreenReaper Studios - WikiFur - Flayrah - Inkbunny
User avatar
GreenReaper
Posts: 23
Joined: Sun Nov 30, 2008 10:01 am
Gender: Male
Sexual preference: bi-mostly straight
Species: Kai Norn (Creatures)
Region: Other
Location: London, UK
Contact:

Re: Inkbunny launches South African cache

#14

Post by GreenReaper »

I've added what's probably the last component for now - a service providing wallpapers locally, which were previously redirected to the Netherlands. As the redirect is cached, you may not see it right away (it probably won't be that noticeable anyway, unless you're scrolling through wallpapers and it has to load them all quickly), but it should take effect some time in the next week.

The Johannesburg cache now serves the islands of Mauritius and Réunion - which are connected to via an underwater cable, and have it even worse than South Africa: it normally takes over 210ms for them to get to the Netherlands and back. Fortunately it's only ~50ms from them to you. Perhaps once Facebook completes its cable, it'll get even faster! (Assuming they don't keep it all to themselves...)
Laurence "GreenReaper" Parry
GreenReaper Studios - WikiFur - Flayrah - Inkbunny
User avatar
Animew
Banter kitty
Posts: 1203
Joined: Wed Jan 21, 2015 12:45 pm
Species: Animu cat

Re: Inkbunny launches South African cache

#15

Post by Animew »

YAYZ! POWER TO THE PAINT RABIT!
i'm kinda curious how many SA artists are on IB other than me tho.
Duck face? i thought they were all just making fart noises when posing for pictures...
User avatar
Rakuen Growlithe
Fire Puppy
Posts: 6427
Joined: Tue Apr 01, 2008 2:24 pm
Gender: Male
Sexual preference: Bi
Species: Growlithe (pokemon)
Region: Gauteng
Location: Pretoria
Contact:

Re: Inkbunny launches South African cache

#16

Post by Rakuen Growlithe »

GreenReaper wrote:
Sun May 17, 2020 11:59 pm
I've added what's probably the last component for now - a service providing wallpapers locally, which were previously redirected to the Netherlands. As the redirect is cached, you may not see it right away (it probably won't be that noticeable anyway, unless you're scrolling through wallpapers and it has to load them all quickly), but it should take effect some time in the next week.
It's cool to see it sometimes loading pages from South Africa. :)
Animew wrote:
Wed May 20, 2020 2:49 pm
i'm kinda curious how many SA artists are on IB other than me tho.
Me. I've even got an Inkbunny sweatshirt.
User avatar
Animew
Banter kitty
Posts: 1203
Joined: Wed Jan 21, 2015 12:45 pm
Species: Animu cat

Re: Inkbunny launches South African cache

#17

Post by Animew »

Rakuen Growlithe wrote:
Wed May 20, 2020 9:15 pm
Me. I've even got an Inkbunny sweatshirt.
you don't count firedog since you hardly ever post stuff, besides as you may have noticed, ive already noticed you when you noticed me... and this has not gone unoticed.

i mean full time artists and the like. i think Sticky is on there but i forget. most SA people go straight for FA for some reason, that or deviant art.
honestly speaking i'm kinda glad tho. inkbunny is like a goldilocks zone for me,
the interface is not too complicated but everything you need is there easily accessable.
the community is just the right size so you dont get lost in a giant shitshow, you actually manage to find cool people on there and everyone is open and understanding.
and the cub stigma keeps away the rif raf. i mean sure theres a few... er... overly enthusiastic furs on there, egged on by the freedom... but the staf actually do their jobs and make quick work of any rule breakers and in my experience there's little to no onsite drama.

also inkbunny generally dont get hacked. not for lack of trying either it seems what with that bigol DDoS recently... cant help but wonder if it was my fault for making so much smut of pinkfong... korea was angry lol.

but yea, if the place gets flooded by normies its just gonna ruin all that.
Duck face? i thought they were all just making fart noises when posing for pictures...
User avatar
GreenReaper
Posts: 23
Joined: Sun Nov 30, 2008 10:01 am
Gender: Male
Sexual preference: bi-mostly straight
Species: Kai Norn (Creatures)
Region: Other
Location: London, UK
Contact:

Re: Inkbunny launches South African cache

#18

Post by GreenReaper »

The cache served 2.5GB of (mostly) thumbnails and site layout images in the last week, from ~200,000 requests by 827 separate IP addresses. Not a huge amount, and some users' access is spread across multiple addresses, but far from nothing. Better get recruiting!

As is common, the pattern of usage is not evenly distributed, with one user appearing to have been responsible for 1/20 of requests and 1/10 of the bytes sent - although, they were new to the site and perhaps performing research, so their usage may be atypical.

One of the high user this week was a returning member and former submitter. Another is an artist with over 250 submissions - Animew (who loaded 2051 images/thumbnails) isn't far off that sum either. Unfortunately without the other artist saying they're from South Africa, I wouldn't feel good revealing it, but I might point them here when I ask them about performance.

So, a few artists, one who appears to be a developer. The majority are regular viewers, several of whom are lurkers with empty profiles. You might think they were inactive. But they're watching. Always watching. 🧐

At current rates it looks like it'll cost ~US$1 a month - reasonable if it helps - wallpapers may be another GB/week, but for that we can redirect to a different sort of hosting with a separate 165MB/day free, so they cost nothing. But it represents a year's promotional pricing; it'd be ten times as much from that provider if SSD, compute instance (RAM, CPU) and IP address were included - or perhaps five times that from another, less featureful provider. Some of that's due to cloud hosts being pricy, or not having much competition, but there are also structural issues like the size of the market, cost of electricity and need to import equipment, run new fibre, etc. So you don't see as many servers in South Africa as other parts of the world. (But still, more than much of the rest of Africa.)
Laurence "GreenReaper" Parry
GreenReaper Studios - WikiFur - Flayrah - Inkbunny
User avatar
Adagio
Warm Hearted
Posts: 3253
Joined: Tue Feb 07, 2012 7:06 am
Gender: Male
Sexual preference: Gay
Species: Snow Leopard
Region: Gauteng
Location: Pretoria, Silverton
Contact:

Re: Inkbunny launches South African cache

#19

Post by Adagio »

Yeah, in South Africa IP Addreses are changed frequently.
On Telkom Mobile, I observed it changing once per second.
Every refresh. But still! That's quite impressive!
User avatar
Rakuen Growlithe
Fire Puppy
Posts: 6427
Joined: Tue Apr 01, 2008 2:24 pm
Gender: Male
Sexual preference: Bi
Species: Growlithe (pokemon)
Region: Gauteng
Location: Pretoria
Contact:

Re: Inkbunny launches South African cache

#20

Post by Rakuen Growlithe »

I guess it's time to forget about the NSA and start worrying about GreenReaper.
User avatar
GreenReaper
Posts: 23
Joined: Sun Nov 30, 2008 10:01 am
Gender: Male
Sexual preference: bi-mostly straight
Species: Kai Norn (Creatures)
Region: Other
Location: London, UK
Contact:

Re: Inkbunny launches South African cache

#21

Post by GreenReaper »

Rakuen Growlithe wrote:
Fri May 22, 2020 7:32 pm
I guess it's time to forget about the NSA and start worrying about GreenReaper.
Absolutely! Although I guess the CLOUD Act means you should worry about the NSA as well.

We don't usually keep access logs on caches, but in this case it was useful to diagnose the performance issue as outlined above. They're unlikely to be used after today unless there's something else to look at, in part because like most cloud things they nickle and dime you to analyze them.
Laurence "GreenReaper" Parry
GreenReaper Studios - WikiFur - Flayrah - Inkbunny
User avatar
Animew
Banter kitty
Posts: 1203
Joined: Wed Jan 21, 2015 12:45 pm
Species: Animu cat

Re: Inkbunny launches South African cache

#22

Post by Animew »

GreenReaper wrote:
Fri May 22, 2020 12:39 am
Another is an artist with over 250 submissions - Animew (who loaded 2051 images/thumbnails)
UwU these numbers please me, makes me feel relevant.
Rakuen Growlithe wrote:
Fri May 22, 2020 7:32 pm
I guess it's time to forget about the NSA and start worrying about GreenReaper.
i'm pretty sure GreenReaper sees everything that ever happens on the internet, firedog. commenting on the most unlikely of posts... being places you really dont expect them to be... somehow omnipresence has been unlocked.
Duck face? i thought they were all just making fart noises when posing for pictures...
Post Reply