Clever Geek Handbook

精英：危险与CosmosDB

图片



o7 cmdr！



在一个温暖的隔离夜里，在“精英：危险”的一个电讯聊天中，就以下话题展开了讨论：哪种类型的恒星最常出现类似地球的行星？



事实是，探索行星是游戏的主要机制之一。在行星有用性的层次结构中，类地行星位于最顶端。但是它们的稀有度也很高。因此，指挥官们想知道：沿着银河系移动时要注意哪些星星？



通过这次讨论，整个项目诞生了，我最终将其埋葬了。不，我们找到了所提出问题的答案。但是由于种种原因，我不喜欢这个项目。经过几个月的拖延，我启动了第二次迭代。这是怎么回事，以及提出的问题的答案-在本文中。



介绍



无论您怎么想，精英人士都会在具有十二个打开选项卡的浏览器中进行很大一部分游戏。我们不会争论它是否正确，但是，毫无疑问，这有它自己的魅力，很多人喜欢它。



, , . , Frontier Developments . , .



? , Log- . .



() ( Windows):



C:\Users\User Name\Saved Games\Frontier Developments\Elite Dangerous\

. . .. PC ( ) — .



, , , . ? - , ( , , , , POI, etc).



PC:



Tool Commander
E:D Market Connector Otis B.
ED-Intelligent Boardcomputer Extension Duke Jones
edce-client Andargor
EDDI VerticalBlank, Hoodathunk, T'Kael
EDDiscovery Robby
Elite Log Agent John Kozak
Elite G19s Companion app MagicMau
Elite Virtual Assistant
Trade Dangerous + EDAPI orphu


. - . , . , , . , 400 (, ! ). , -.



, . ( , ). : , , (, ) .. , , , , — EDSM EDDB.



, , — . .





EDSM , . , , . , , — JSON 10. - - . , , - , . , ! .



图片



% . 6 . .



, , . , , !



, , , - ( , ). EDSM, EDDB, etc. — EDDN. , , , .



, EDDN ? ...





EDDN



EDDN — :



The Elite: Dangerous Data Network is a system for willing Commanders to share dynamic data about the galaxy with others.

By pooling data in a common format, tools and analyses can be produced that add an even greater depth and vibrancy to the in-game universe.



EDDN is not run by or affiliated with Frontier Developments.

( ).



HTTP Endpoint https://eddn.edcd.io:4430/upload/ ( ).



. ZeroMQ tcp://eddn.edcd.io:9500. .



( ):



图片



, . ? , . , ( ) ! , - .





, :



图片



— Microsoft Azure. - . — Azure, dotnet core/standard



--- , .



, :



1. Message Distributor



EDDN Channel . ( schemaRef , ) — . EDDN 5:





. ? , ( ), , . journal, .. ( ). , , .



, MessageDistributor EDDN . .



2. Azure Storage Queue



( Message Processor , ). , . Storage Account ( ) connection string ( , Azure AD, ). Storage Account ( — , , ).



Azure Storage Emulator Azure Storage Explorer


3. Message Processor



, , . Azure Function App.



, - ? .



, Azure Function WebJob ( , ), . , Azure Function Runtime ( , ) — — , — . , .. — .



, , (Scale Out), , , - , CPU/Memory consumption, etc.



, (in/out bindings). , , QueueTrigger — (, dotnet, ). CosmosDBTrigger . ( ). , , CosmosDB ( ) . - , , : db client ( in-built DI, ). , , queue .



QueueTrigger. , — (invisible) . , 30 ( ). 30 — DequeueCount 1. ( -> visible state on, ++DequeueCount). DequeueCount = 5 ( ), -poison. journal, 5 , journal-poison. ( - ). , . .

: WebJob AppService. — Function App. : , .



. : , , 2 . 2 , , . , (upsert — update || insert) , , ;-) 2 , , . , . , . , , , . , - (, , CosmosDB ACID compliant).



IMHO: , . . 1. , — . (, ) .



Cosmos DB Optimistic Concurrency, . . .



4. CosmosDB



. , ( ) , . , .. .



4: Signals, Systems, Stations, Bodies. , journal ( , ? -).



, journal Event, : CarrierJump, Docked, FsdJump, Location, SaaSignalsFound, Scan ( ). , — . : — . .



CosmosDB. ( , ) Request Unit per second — RU/s. .. RU/s . 2 : Provisioned throughput Serverless ( , ). , Provisioned RU/s ( ), Serverless, RU/s , , .



Provisioned RU/s. , RU/s 400, 10% 1000% (40-4000 RU/s)



RU/s Provisioned mode — 400. , , , 250 RU/s 150 , Serverless .



, ( CosmosDB). : CosmicClone. Serverless . 2 . Provisioned Mode , , .



, zero downtime "" — , MessageDistributor , . , . . .



CosmosDB

, , : EDDN , .





, React Native — dotnet, . .



: EDDNConsumer. , , .



:



  • EDDNConsumerCore — MessageDistributor. EDDN
  • EDDNModels — EDDN
  • JournalContributor — MessageProcessor journal
  • SharedLibrary — . .


EDDNConsumerCore



dotnet core 3.1 . Main DI. HostedService ConsumerService



services.AddHostedService<ConsumerService>();


nuget- ( ): NetMQ — ZeroMq Ionic.Zlib — NetMQ



StartAsync, — NetMQRuntime ClientAsync ( , ):



private async Task ClientAsync()
{
    var utf8 = new UTF8Encoding();
    using (var client = new SubscriberSocket())
    {
        client.Connect(_eddnClientSettings.ConnectionString);
        client.SubscribeToAnyTopic();
        while (true)
        {
            try
            {
                (var bytes, _) = await client.ReceiveFrameBytesAsync();
                var uncompressed = ZlibStream.UncompressBuffer(bytes);
                var result = utf8.GetString(uncompressed);
                await _messageDistributor.DistributeAsync(result);
                _logger.LogInformation(result);
            }
            catch (Exception ex)
            {
                _logger.LogError(ex, "Error reading message queue");
            }
        }
    }
}


, ( , 1 ), _messageDistributor , , . .



MessageDistributor DistributeAsync:



public async Task DistributeAsync(string message)
{
    try
    {
        using var stringReader = new StringReader(message);
        using var jsonReader = new JsonTextReader(stringReader);
        var result = _serializer.Deserialize<Entity<BaseMessage>>(jsonReader);
        var queue = await _messageQueueFactory.GetQueueAsync(result);
        await queue.SendMessageAsync(message.Base64Encode());
    }
    catch(Exception ex)
    {
        _logger.LogError(ex, "Error distributing message");
    }
}


, , MessageQueueFactory. — . : . , schemaRef header ( test_data.json ). , , .



, ( — JSON) , . \\ , — , , null default. ( ) , POCO , , , , - . .



, , json- NewtonsoftJson ( json, ). Missing Member



_serializer.MissingMemberHandling = MissingMemberHandling.Error;
_serializer.Error += _serializer_Error;


, Application Insights, Notification Alerts . , , Header. . , .. , missing property. , , .



, header , . , ( ):



public class MessageQueueFactory : IMessageQueueFactory
{
    private readonly StorageAccount _storageOptions;
    private readonly QueueMapping _queueMapping;
    private readonly IDictionary<string, QueueClient> _queues = new Dictionary<string, QueueClient>();

    public MessageQueueFactory(
        IOptions<StorageAccount> storageOptions,
        IOptions<QueueMapping> queueMapping)
    {
        _storageOptions = storageOptions.Value;
        _queueMapping = queueMapping.Value;
    }

    public async Task<QueueClient> GetQueueAsync(Entity<BaseMessage> entity)
    {
        if (_queueMapping.TryGetValue(entity.SchemaRef, out var queueName))
        {
            if (!_queues.ContainsKey(queueName))
            {
                var client = new QueueClient(_storageOptions.StorageConnectionString, queueName);
                await client.CreateIfNotExistsAsync();
                _queues.Add(queueName, client);
            }
            return _queues[queueName];
        }
        throw new ArgumentException($"Queue {entity.SchemaRef} has not configured", nameof(entity.SchemaRef));
    }
}


, appsettings json :



{
  "QueueMapping": {
    "eddn.edcd.io/schemas/journal/1": "journal"
  }
}


( ):



"QueueMapping": {
    "eddn.edcd.io/schemas/journal/1": "journal",
    "eddn.edcd.io/schemas/blackmarket/1": "blackmarket",
    "eddn.edcd.io/schemas/commodity/3": "commodity",
    "eddn.edcd.io/schemas/outfitting/2": "outfitting",
    "eddn.edcd.io/schemas/shipyard/2": "shipyard"
  }


Dictionary<string, string> schemaRef. .



1 _queues — QueueClient, . , . , . round trip , . , , . , .



, — , — . . MessageDistributor, .



. 3 . .



EDDNModels



POCO . , . , JournalMessage:



[JsonProperty("id")]
public override string Id
{
    get => Event switch
    {
        JournalEvent.FsdJump => StarSystem,
        JournalEvent.Scan => BodyName,
        JournalEvent.Docked => StationName,
        JournalEvent.Location => BodyName,
        JournalEvent.CarrierJump => StarSystem,
        JournalEvent.SaaSignalsFound => BodyName,
        _ => $"UnknownID_{Guid.NewGuid()}"
    };
}


Id, , CosmosDB. ORM. , journal ? , . Event Id. - , - , - .



. : , , , , , , , , , \ .. ( JournalMessage). , , .. POCO — .



.



JournalContributor



"" journal — Azure Function App. Function App dotnet core class library , , . FunctionName . : — Function App ( ). - — . , . . , , ( ).



, :



public class JournalContributor
    {
        private readonly IEventTypeProcessorFactory _eventTypeProcessorFactory;

        public JournalContributor(IEventTypeProcessorFactory eventTypeProcessorFactory)
        {
            _eventTypeProcessorFactory = eventTypeProcessorFactory;
        }

        [FunctionName("ContributeJournal")]
        public async Task Run(
            [QueueTrigger("journal", Connection = "AzureWebJobsStorage")]
            Entity<JournalMessage> myQueueItem,
            ILogger log)
        {
            try
            {
                var eventProcessor = _eventTypeProcessorFactory.GetProcessor(myQueueItem.Message);
                await eventProcessor.ProcessEventAsync(myQueueItem.Message);
            }
            catch(Exception ex)
            {
                log.LogError(ex, $"Error processing queue item: {JsonConvert.SerializeObject(myQueueItem)}");
                throw;
            }
        }
    }


QueueTrigger ( ). , Event … .



, , . — , dotnet . Startup Configure, FunctionStartup (assembly):



using JournalContributor.Settings;
using Microsoft.Extensions.Configuration;
using System.IO;

[assembly: FunctionsStartup(typeof(JournalContributor.Startup))]

namespace JournalContributor
{
    public class Startup : FunctionsStartup
    {
        private IConfigurationRoot _functionConfig;
        private readonly string COSMOS_CONNECTION_STRING = Environment.GetEnvironmentVariable("CosmosDBConnectionString");
        private readonly string ENVIRONMENT = Environment.GetEnvironmentVariable("ASPNETCORE_ENVIRONMENT");

        public override void Configure(IFunctionsHostBuilder builder)
        {
            _functionConfig = new ConfigurationBuilder()
                .AddJsonFile(Path.Combine(builder.GetContext().ApplicationRootPath, "appsettings.json"), optional: true, reloadOnChange: true)
                .AddJsonFile(Path.Combine(builder.GetContext().ApplicationRootPath, $"appsettings.{ENVIRONMENT}.json"), optional: true, reloadOnChange: true)
                .Build();

            builder.Services.AddSingleton<CosmosClient>(factory => new CosmosClient(COSMOS_CONNECTION_STRING));
            builder.Services.AddSingleton<IEventTypeProcessorFactory, EventTypeProcessorFactory>();
            builder.Services.AddTransient<FsdJumpProcessor>();
            builder.Services.AddTransient<ScanProcessor>();
            builder.Services.AddTransient<DockedProcessor>();
            builder.Services.AddTransient<LocationProcessor>();
            builder.Services.AddTransient<SaaSignalsFoundProcessor>();
            builder.Services.Configure<CosmosDbSettings>(_functionConfig.GetSection("CosmosDbSettings"));
        }
    }
}


DI . , CosmosClient : (CosmosDB). ? . 2 CosmosDB — , . , .



. , , , switch :



public IEventTypeProcessor GetProcessor(JournalMessage journalMessage) => journalMessage.Event switch
        {
            JournalEvent.FsdJump => _serviceProvider.GetService<FsdJumpProcessor>(),
            JournalEvent.Scan => _serviceProvider.GetService<ScanProcessor>(),
            JournalEvent.Docked => _serviceProvider.GetService<DockedProcessor>(),
            JournalEvent.Location => _serviceProvider.GetService<LocationProcessor>(),
            JournalEvent.SaaSignalsFound => _serviceProvider.GetService<SaaSignalsFoundProcessor>(),
            JournalEvent.CarrierJump => _serviceProvider.GetService<DockedProcessor>(),
            _ => throw new ArgumentException($"Unknown Event {journalMessage.Event}", nameof(journalMessage.Event))
        };


, , . ScanProcessor, .. 2- :



public async Task ProcessEventAsync(JournalMessage message)
        {
            var existingItem = await message.CheckIfItemExists(_bodiesContainer, _options.BodiesCollection.PartitionKey);
            if (existingItem == null)
                await _bodiesContainer.UpsertItemAsync(message);
            else
            {
                //Basic scan type contains less information then others.
                //We`re skipping item upsert if remote item has scan type higher then Basic
                //We`re also skippings if remote item scan type is Detailed (as it`s a maximum info scan)
                //Will update item if both (remote and current) have scan type Basic
                //And will upsert item in case of current item have higher scan type then remote
                if ((message.ScanType == ScanType.Basic && existingItem.ScanType != ScanType.Basic) ||
                    existingItem.ScanType == ScanType.Detailed)
                    return;
                else
                    await _bodiesContainer.UpsertItemAsync(message);
            }
        }


, , ( !) , , , , . . . , . , : , Bodies Id . , .



UpSert, Insert, .. concurrency : Item with such Id already exists ( - ). , , Optimistic Concurrency . . upsert.

, ScanType. Detailed — . Basic — — . Basic — . .



. , 3 , MessageProcessor journal .





, . , Azure WebJob EDDN , Storage Account , Azure Function App Cosmos DB . Application Insights (, -, , ). , .



图片



continuous, .. .



![image](img src="https://habrastorage.org/webt/i_/5u/zq/i_5uzqxkanblmotxacqhtwnp_wq.png)



, , schemaRef . , .



图片



Stream-log : , , … .



图片



. , , .



图片



RU/s . — Provisioned Mode 400 RU/s. , . ~105RU/s.



图片



. 145 . EDSM, .



图片



. Check.



. . .





. . - .



45 . , MSDN , 45 . . , . . 45 — 1.5 . , — .



— . - — , - — "" . .



图片



. ~10 . appinsjournalweprivate . Function App. — , . \. :



图片



( ), host.json JournalContributor:



{
  "version": "2.0",
  "logging": {
    "applicationInsights": {
      "samplingExcludedTypes": "Request",
      "samplingSettings": {
        "isEnabled": true
      }
    },
    "logLevel": {
      "default": "Information",
      "Host": "Error",
      "Function": "Error",
      "Host.Aggregator": "Information"
    }
  }
}


:



图片



, ( 21 ):



图片



( 193 ). - / . 21 1.75 . — appinsweprivate — WebJob. 0.98 0.19 21 .



cdbweprivate — CosmosDB Provisioned Mode 400 RU/s. . cdbslweprivate — Serverless mode CosmosDB. 4.47 .



— laweprivate. Log Analytics Workspace, , , .. CosmosDB. , , , . .



, 175,8 . , .



, . -. appinsights — 3 . , . , % , . - , , 3 — ( ).



-. CosmosDB RU/s ( , ) RU/s (Pricing), :



图片



EDDN. , , Elite: Dangerous. - " " — . , , DLC Odyssey , . .



, Function App Storage Account, , .



, . — .



. .





.



" " " ", . - , . =) , -, ...



— . , - , 145 , EDSM 57 . - - , .



, , , . . " " — .



图片



, — . ? . , , EDSM . , JSON — . — ( ). RU/s , , . .



CosmosDB — Jupyter Notebooks. C#, Python, - . , .



, , . ( , Sol [0,0,0]:



图片



? :



图片



5 :



图片



EDSM, - , . , , — - . , , .



CosmosDB, Azure Function Change Feed ( ), .



— CosmosDB .



对于那些坚持到底并仍对在精英中寻找类似地球的行星感兴趣的人，这是您的答案（结果基于大约六个月前从完整的EDSM转储获得的数据）：



图片



F，K，G，A型以及中子（由于某种原因）构成了前五种类型的恒星，飞行员可以从中找到类似于地球的行星。好吧，这些是结果¯\ _ （ツ） /¯



好。就这样。



安全飞行，cmdr！



More articles:


All Articles