Large Scale Server Operations
in Mobage's Social Games
with Perl

Ryosuke IWANAGA - riywo @DeNA
YAPC::Asia 2011
October 14, 2011

Self Introduction

• Name: Ryosuke IWANAGA
• ID: riywo
• Work: DeNA Co., Ltd. (Apr 2009~Present)
• Position: Mobage DataStore Team Leader
• Job: Mobage's Server Administration/Operation
• Recent work:
  • writing: SoftwareDesign 2011 Sep
  • attending: MySQL Conf. 2010, Velocity 2011

Agenda

• System architecture of Mobage
• one of the biggest mobile web services in Japan
• Case studies on DeNA's Mobage SocialGame
• /** not OpenPlatform **/

Why am I here today?

• I watched an impressive presentation from YAPC::Asia 2010

source: 省サーバ運用

Why am I here today?

• nekokak disregarded "large scale" ;p
• But he said great things like "I think it'd be better if more infra engineers came out in public!"
• I was encouraged by this phrase
• Then, in Mar 2011....
• nekokak joined DeNA! ｴｯ(ﾟДﾟ≡ﾟДﾟ)ﾏｼﾞ?
• DeNA is "large scale", right? :)
• I felt this must be a fate, so I decided to talk about "large scale operations" against nekokak!
• And he advised me! Thanks nekokak!

System Architecture of Mobage SG

System Architecture of Mobage SG

System Architecture of Mobage SG

• AS38636
• WEB/APP servers
• WEB is Apache / APP is written in Perl
• APP is running as FastCGI ExternalServer
• DNS servers
• Internal DNS to access the other servers
• Cache servers
• Consistent Hashing or Redundant
• DB servers
• MySQL Master/Slaves

System Architecture of Mobage SG

• That's all! :)
• Very simple / standard web system
• Mobage Portal and SocialGame have the same architecture
• Without backend job servers?
• Some jobs are backend, but main jobs are frontend
• a bit different from Open Platform
• We tried to scale out these components

Case Studies in DeNA's Mobage SG

Case Studies in DeNA's Mobage SG

• Started was Sep 2009...
• DeNA released some SocialGames on Mobage
• Settlerin, Kaizoku Treasure, Kaito Royal, etc.
• At first, they started small
• All games shared only 1 master DB and some slaves
• Then, the changes happened...

Case Studies in DeNA's Mobage SG

• They grew up larger and faster than we expected
• Everything was not enough
• Network equipments capacity
• Leased lines bandwidth
• DataCenter racks
• Of course, servers... X(
• And we encountered many difficult performance problems
• I'll talk about some approaches from the server side

Case Study 1: Application Tuning

Case Study 1: Application Tuning

Point:

• 2 access interfaces

Case Study 1: Application Tuning

• App servers connect many times to other servers
• In large scale, the less the number of connections, the more stable the system
• At that time, the Mobage application called for DNS resolver for every DB connections
• If the network went over capacity, some packets were lost, then DNS was heavily damaged (because of UDP)
• -> we made an application name resolving module

Case Study 1: Application Tuning

Case Study 1: Application Tuning

• We built/introduced HandlerSocket interface(MySQL plugin)
• This is another access interface of MySQL
• Faster than SQL interface, and also Memcached server!
• Don't care about consistency between database and cache. Just access the same data using a different interface
• We replaced the method of getting user data from Memcached to HS
• Became faster, and APP's cpu usage decreased
• C::M::Fast cost too much when calculating consistent hash

Case Study 1: Application Tuning

• Common points of Resolving Module and HandlerSocket
• 2 access interfaces
• Resolving Module: DNS and MySQL
• HandlerSocket: MySQL and HS protocol
• New interface is easy to introduce
• Incrementally test/measure/introduce
• Old interface modules are also enabled
• great! :)

Break Point

• 19/41
• about 15 min?

Case Study 2: Database Tuning

Case Study 2: Database Tuning

Point:

• Exposed problems

Case Study 2: Database Tuning

• After release, Kaito Royal immediately grew
• We divided DBs many times

Case Study 2: Database Tuning

• Normally Kaito users battle others and bet treasures for/from their collection
• The treasure is dropped in missions, and missions vary depending on the user's level
• Users search the one who has the targeted treasure, and battle
• So, users will battle other users of about the same level users

Case Study 2: Database Tuning

• 3 months after the release, Kaito Royal started the first "Event"
• In the Event, All users got the same treasures from all of the missions
• And these treasures were limited time items
• Users searched and battled so many people, so many times
• Can you imagine what happened?

Case Study 2: Database Tuning

• First: searching requests became too slow

• Index

PK: user_id,treasure_id
k1: treasure_id,flg,level,num

• Search Query

select user_id from t
where treasure_id=10 and flg=0
and level>=20 and level<30
order by num desc limit 200

Case Study 2: Database Tuning

• At first, MySQL narrowed down the rows using "treasure_id=10 and flg=0"
• Then, sorted "num", and checked "level" condition until it found 200 rows
• This worked for no-event time because a treasure was held by users of about the same level
• Narrowing down was fine and finding the same level users was easy
• But in the Event, many users held the same treasure...
• So, narrowing down was bad, and finding users was difficult, especially for high level users

Case Study 2: Database Tuning

• We changed the "level" column
• Preparing a distribution of the levels in advance
• set "level-class" number to "level" column(1-10=1,11-20=2,...)
• If "select ... where treasure_id=10 and flg=0 and level=2 order by num" was used, MySQL could narrow down the rows better
• If better searching is required, you can use Redis or something :-P

Case Study 2: Database Tuning

• Heavy battling caused a "Lock wait timeout" problem
• Kaito had many DB masters
• If Deadlock happened for 2 or more masters, the MySQL couldn't detect it, so MySQL waited while innodb_lock_wait_timeout
• We decreased the parameter to 5 (if you set it for less than 3, it can be affected by TCP SYN retransmission..)
• And we changed the application
• When locking the records, sort some id(ex. user_id) to avoid Deadlock from happening

Case Study 2: Database Tuning

• These 2 problems had not appeared before starting the Event
• In the Event, user activity was changed, so problems were exposed
• When you plan Events or new features, imagine not only new schemata but also existing tables

Break Point 2

• 31/41
• about 30 min?
• Ok! The rest is "spiritual"

Case Study 3: DevOps Working

Case Study 3: DevOps Working

Point:

• Ops are project member/manager

Case Study 3: DevOps Working

• When the Event was released, there was a DevOps problem
• We(Ops) were not given anything about the Event in advance X(
• If information had been shared ahead of time, we could advise and add servers before the Event started
• **Notice** we can't predict all problems
• But knowing the system and the release timing will be useful for troubleshooting
• The Project Manager should consider the time of sharing Ops

Case Study 3: DevOps Working

• And in the sharing, each time, Ops should think "Are this event specs right?"
• Sometime a little compromise make us happy
• Especially "Realtime" :)
• Ops should advice on such things
• Not only Dev but also Ops are Team Members :)
• Never be just men who follow the command

Case Study 3: DevOps Working

• In SG, there were so many tuning points
• Even if Ops list up all the points and show them to the Dev people and say "These are bad. Please modify these points"...
• They may be confused because they can't decide which are more important
• Also Ops can't determine the difficulty of modifying the points
• Which is the better approach?

Case Study 3: DevOps Working

• Example: Ops found many bad SQL on a DB server
• Ops can summarize the total query execution time
• It's easy using pt-query-digest, or DBILog at DeNA
• Ops should check top 10 queries
• explain, index of cause
• purpose, cacheable reading of the source code
• Ops should consult with Dev about a few queries which must be modified for the server resource and the response time
• I think it is important that Ops choose a few significant queries and think about them together with Dev

Case Study 3: DevOps Working

• Because Dev has many development tasks
• Performance tunings are additional tasks
• I think Ops should be responsible for the tunings
• priority, how to resolve it, checking the deadline, effect
• Ops should think about the above and get the man-hours from Dev
• This is a small software development project! Ops should behave like PM
• I did many projects like Query Tuning, and Data Purge :) Thank Dev team!

At the end

At the end

• I talked about the below topics
• 2 access interfaces
• Exposed problems
• Ops are project member/manager
• Operation is such creative work!
• Large scale is challenging
• DeNA is an exciting company
• Why don't you join us? Please talk to me!

Thank you!

• using html5slides. Thanks!