Mirai + Megane
Over the past many months I've been working on a rewrite of Mirai, moving to a clustered architecture. Previously Mirai ran entirely in one process, and this was incapable of handling the amount of data received from Discord. To solve this I created megane, a Discord bot framework that splits a bot up into multiple clusters to make use of threads (something NodeJS doesn't do). Here you can see the results of the first live test of megane:
Mirai Bot CPU usage with mirai-bot-core. Notice how often the CPU core reaches 100% utilization, blocking things like commands and causing disconnects.
Mirai Bot CPU usage with megane. Notice that each cluster has its own core, spreading computation out so one task can't block the entire bot.
Removal of Presence
Presence data makes up the large majority of all events from Discord. If you don't know what presence data is, it's basically your current state in Discord. It's your activities (game status), online status, username, avatar for every guild (server). Every time one of these things changes, a presence update is sent to every client in every guild. So imagine if Rythm, which is in 12M+ guilds, updates it's game/status. That means for each of those 12M+ guilds every member that's connected will get a presence update. You can imagine what this means for bots that are in a large amount of guilds.
As of now, Mirai receives around 200-250 million presence events per day. If you do the math that's around 2,500 presence updates per second.
With the addition of intents to the Discord API presence updates are largely disabled for bots. Getting access to them requires manual whitelisting which is not easy to get. Luckily Mirai hardly makes use of them after recent changes, so the only thing lost is a few pieces of info in the m.info command. Without needing to process these events Mirai would probably run without issue even on the current design. Here is the most recent CPU usage graph from the second megane test without presence:
Notice how it doesn't even seem like anything is running.
Unfortunately with intents also comes a huge change to the way guild member chunks work. If you don't know, guild member chunks are a resource that bots request to receive every member in a guild (if they need it). Before intents guild member chunks could be requested all at once on connecting. However, without the guild presence intent this is limited to one guild per request. With a total shard websocket hard limit of 120 requests per minute this obviously causes issues. Eris, the api library megane uses, does not even support this. I did the math, and found out that for a shard with 2,000 guilds (the max is 2,500) it would take 20 minutes to cache every guild's members at 100 guilds per second. I obviously didn't want to further delay the ready event by 20 minutes, so my only option was to write my own caching logic. I added a custom member caching system to megane that caches guild members after the cluster is ready. This allows most of the bot to operate while the parts that need a complete cache are disabled until everything is ready for them. You'll notice this indicated in Mirai by the "starting..." status.
How Megane Works
Megane works by splitting Discord shards up into clusters and running those clusters in separate processes. By default the number of clusters is the number of CPU cores on the server. This allows it to make use of every core, while keeping overhead down by not creating too many unneeded processes. A simple way to think of this is that it's assigning each cluster to a CPU.
Services are also a new concept in megane. Services are programs in megane that run in their own process. Some examples in Mirai are the Twitch, anime, and Patreon service. Each of these services are used by all of Mirai, but only one instance exists which every client can communicate with. Running these in their own process prevents things like Twitch data updates from blocking a thread and causing issues with the bot.
For a visual illustration of how this is all laid out you can use this poorly-made diagram I made:
Notable Changes in v5
Here you can find a reduced changelog with less important changes removed. For the full v5 changelog scroll to the next section.
- Command plugins are now known as command categories, and the normal category is now known as the general category
- Markdown escaping is now done in more messages
- Improved the live stats message in the official Mirai server, and added guild count updating for more bot lists
- Some features, such as twitch notifications, are disabled until all members are cached
- Simplified error messages for almost all commands, and added more error handling
- Improved responses for almost all commands when something is done wrong
- Improved descriptions for almost all commands
- Fixed Twitch features not working
- Twitch data updates run faster, reducing the delay in Twitch notifications
- Upgraded Patreon code to use the new API, and added support for webhooks for instant updates
- Fixed animated emotes not working in welcome messages and MOTD messages
- Fixed anime data updates missing data due to a logic error
- Replaced the
m.ping command with a new command, m.status which shows much more info
- Fixed an option being listed incorrectly in the
m.facts commands, and allowed omitting options to get random results instead
- Increased the maximum giveaway winners to 25 at once
- Significantly improved the
m.info command responses
- Changed the
m.reddit command to allow "r/" and embed links by default
- Increased the reminder limit from 8 to 20, but required the list command to be used in a DM when having more than 8 reminders
- Increased the reminder time limit to one year and added year recognition
- Removed the
m.rip command
- Redesigned the response for
m.urban to include more info and format links
- Added bot permissions checks and an indication of if the user could be DM'd to the
m.ban, m.kick, m.softban, and m.forceban commands
- Changed the
m.permissions command to instead show denied permissions if the --all option is used, and to format permissions in a user-friendly way
- Fixed the user filter in the
m.prune command and added discord.com recognition to the invites filter
- Fixed the airing command using UTC instead of JST for channels
- Added Japanese title match checking in the anime, character, and manga commands
- Fixed the
m.ratewaifu command not ever using truly random ratings, removed partial username/nickname matching, and made text less gender-specific
- Fixed Mirai not handling voice channel disconnects and channel switches properly
- Changed the
mm.stop command to require the "Move Members" permission
- Fixed the
m.skip command telling the user the wrong command to create a player
- Fixed the
m.pause command having the wrong prefix ("p" instead of "ps")
Full Changelog
Core Changes
- Bluebird promises have been removed for native promises
node.blocked stat removed- Dependencies
- Updated all compatible dependencies
- Added missing
eslint dependency
- Eris
- Disabled compression
- Added configuration for
allowedMentions
- Added intents
- Command base
- Changed to use async/await
- Can now require multiple permissions
- Missing permissions are now formatted in a friendlier way: "manageMessages" -> "Manage Messages"
- Made escaping markdown remove block quotes too
- Command plugins are now called command categories
- The describe command now includes the command category when naming the command if it doesn't have a description, if used
- MongoDB
- Re-wrote to be required in all files needing it, instead of passed around to classes, as a class
- Removed the pointless method wrappers, and the useless cache
- Changed code using deprecated methods like update() and remove()
- Changed the config file to a JS file for easier use
- Changed ESLint config to a JS file and modernized/updated the configuration
- Removed bodyparser since it is bundled with express now
- Created a new logging system for sending errors and warnings to Sentry
- Improved statistics tracking
- Set "starting..." status until all members are cached
- Bot list stat updating
- Added back updating to
discord.bots.gg and top.gg
- Changed updating to every 4 minutes
- Member caching for guilds is now done after the ready event because of Discord API changes. This is estimated to take up to 20 minutes after a cluster is ready, so features that do not require a complete cache are enabled, while those that do are disabled until caching is complete.
Service Changes
- Twitch service
- Now runs in it's own process to not block other code from executing
- Now uses the
twitch package instead of custom code for handling interactions with the Twitch API
- Now authenticates with OAuth, as required, giving a much higher rate-limit
- Simplified function to fetch stream data for notifications
- Improved code deciding whether to display "Unknown game" or "No game set"
- Simplified code for updating database entries for channels/users
- Slightly improved notification message content
- Patreon service
- Upgraded to v2 of the Patreon API
- Added support for webhooks to instantly update data when pledges are created/updated/deleted
- Fixed the
removeOldPatrons() function not setting previous patron's keys to expire, and changed it to delete the keys to be consistent with other functionality
- Improved the wording for notifications about keys
- Event notifications
- Fixed support for animated emojis
- Refactored code
- Slightly changed event titles
- MOTD service
- Only one database op is done for each guild. Previously one was done for every channel (with a MOTD update)
- MOTD events are sent one after the other to prevent the tasks from blocking when being sent all at once
- Slightly better error handling
- Fixed support for animated emojis
- Anime service
- Fixed airing notifications having different external links from standard anime info responses
- Fixed messages potentially showing "null" as the total episode count when it was unknown
- Fixed not getting all pages of airing anime at startup (got first page twice and didn't get last page)
Command Changes
- Simplified error messages for all commands
- Improved error handling for commands missing it
- Twitch commands
- Improved failure and help messages for all commands
- Changed references of "Status" to "Title"
- Improved the stream info section of the channel command response
- Improved descriptions
- Added "followed" prefix to the following command
- General commands
- Renamed "Normal" category to "General"
- Added the aliases "8" and "ask" to the 8ball command
- Updated the avatar command description to list the correct maximum avatar size
- Catgirl command
- Removed the "nyan" and "nyaa" aliases from the catgirl command
- Updated the description to list the website used
- Changed the cooldown to not activate when too many tags are submitted
- Fixed spelling issues in the choose command
- Improved wording in responses for the coinflip command
- Status command
- Replaces the ping command
- Has aliases "ping" and "shard"
- Shows debug information about the current cluster and shard for the guild
- Improved responses and error handling in the crypto command
- Dice command
- Improved description
- Changed the limit to show individual rolls to include 10 rolls
- Removed useless formatting call
- Currency command
- Improved description, and fixed the args being positioned incorrectly
- Improved failure messages
- Facts command
- Improved description, and fixed an option being incorrect
- Improved handling for omitted options, making them function the same as typing "random"
- Improved the description for the fortune command
- Giveaway command
- Improved description
- Improved some strings
- Increased the maximum winners to 25
- Improved the responses for when a giveaway is ended automatically with no winners
- Pro command
- Removed reference to feature that will likely never be added in the info/tldr sub-command
- Improved the wording for many messages
- Info command
- Improved formatting
- Increased cooldown to 3 seconds
- Improved failure responses
- Guild info
- Updated channel related items to use current channel types
- Created security field and moved notification and mfa items to it
- Added explicit content filter item
- Updated and improved values for the notifications item
- Moved "Other" field to the end
- Added items for public/partnered/vip server settings
- Removed online member count since status is not received anymore
- Improved formatting for channel and member counts
- Channel info
- Moved everything to the description
- Fixed NSFW showing for voice channels
- Added slowmode item
- Added user limit item for voice channels
- Added category item
- User info
- Moved everything to the description
Status
- Fixed statuses
- Added support for Competing status
- Changed member status text
- Added emoji to status text if set
- Escaped markdown in status
- Increased role list text limit to 400 from 250
- Changed timestamp items to be on one line
- Added "boosting since" item
- Removed status since it is not received anymore
- InRole command
- Removed "members" alias
- Improved description
- Improved failure messages
- Escape markdown
- Map command
- Improved description and failure responses
- Escaped markdown in the Random command response
- Refactored the RandomComic command
- Reddit command
- Improved description
- Embed links by default
- Allow using "r/" in addition to "/r/"
- Refactored
- Added formatting to post scores
- Fixed errors from trying to format non-posts (The api returns subs instead of posts when the sub is not found)
- Reminder command
- Increased the limit to 20 reminders (from 8)
- Improved responses
- Fixed logic for max input tries stopping one try before it should
- Removed reminder list when running the "remove" subcommand without any text
- Change the list subcommand to cut off text on long reminders and to only allow the command to be run in DMs when the user has more than 8 reminders
- Added recognition for "mns" and "mn" as months
- Fixed the possibility of raw errors being sent to users
- Changed the maximum reminder time to one year, and added year recognition
- Removed references to removing reminders without a message, as it has not been possible to create one for a long time
- Removed the RIP command
- RockPaperScissors command
- Improved description
- Added more responses
- Strawpoll command
- Added note about the vote command in the description
- Improved responses
- Fixed extra character in the poll name when a title was not specified
- Urban command
- Improved description
- Designed a more detailed response
- Vote command
- Improved description
- Added prefix "v"
- Changed the name of votes with no topic to "Untitled Vote"
- Improved responses
- Made sure vote results are shown even if an error occurs removing a vote in the database
- Refactored the code
- Weather command
- Improved description
- Improved failure messages
- Moderation commands
- Ban, kick, and softban commands
- Improved description
- Improved responses
- The message sent in the server now includes the reason and whether the user could be DM'd
- Added bot permission check
- Force ban command
- Improved description
- Improved incorrect usage response
- Added bot permission check
- Permissions command
- Improved description
- Changed denied permissions to instead show if
--all is used in the command
- Converted permission format "manageMessages" -> "Manage Messages"
- Escaped markdown in username/nick
- Slightly improved responses
- Prune command
- Improved description
- Fixed aliases not being listed
- Improved failure messages
- Fixed mentioning a user for the
user option in the prune command
- Fixed the invites option not detecting discord.com links
- Fixed shorthand option parsing checking for unreachable option and not removing duplicates
- Role command
- Improved description
- Improved failure messages
- Added alias "roles"
- Escaped markdown on role names
- Improved the description of the SetAvatar command
- Fixed link suppression not working with the SetGame command
- Anime commands
- Airing command
- Improved description
- Improved text and formatting of command responses
- Removed useless code, cleaned up other code, and improve error handling
- Fixed messages for channels using GMT instead of JST
- Anime, manga, and character commands
- Improved help text and failure responses
- Added check for exact matches with the native title (Japanese characters)
- Made sure command response will never have an undefined title
- Ratewaifu command
- Made text less gender-specific
- Improved help and failure messages
- Removed useless double rng
- Fixed the condition to use a non-seeded rating always being false, and lowered the chance to 20%
- Changed user matching to only match ids and exact name or nickname matches (not case sensitive)
- Music commands
- Mirai now properly handles being disconnected from a voice channel, and doesn't break until restart
- Mirai now properly handles being moved to different channels
- Changed command references to use the short prefix "mm."
- Bind, volume, stop, skip, resume, pause, and queue commands
- Improved description
- Improved responses
- Improved responses for the join command
- Improved responses for the nowplaying command
- Changed required permission to "Move Members" for the stop command
- Fixed a response telling the user to use the wrong command to create a player in the skip command
- Fixed the pause command having the wrong alias ("p" instead of "ps")
- osu! commands
- Profile, Recent, Signature, and TopScores commands
- Improved description and missing user message