# X-Road

# 101

ee-test/GOV/70007446/hksos/ambulanceResourcesV2/v1

keskkond/asutuse-tüüp/reg-kood/alamsüsteem/teenuse-nimi/teenuse-ver

# Attachment(s) / Manused

soapi manust pole kunagi salvestatud,ainult soap body, või rest body

soap manus ei ela mitte kusagil seda ei salvestata, ja pole kunagi salvestatud. See on by design nii ,et soapi manusest jääb vaid hash signatuuri sees <span title=":slight_smile:"><span class="rcx-message__emoji emojione emojione-people _1f642 :slight_smile:">🙂</span></span>

kui turvik saadab sõnumi, siis võib olla korraks paneb tmp kausta, ja siis suunab kasutajale edasi ja see manus kustub igavesti

# Errors

##### Server.ClientProxy.SslAuthenticationFailed  
Service provider did not send correct authentication certificate

Klient annab teada et teenuse turvikuga ei õnnestu handshake teha sest ei tule teenusepakkuja poolt auth serti

##### Server.ServerProxy.ServiceFailed.HttpError&gt;  
Server responded with error 403: Forbidden

Võrgumure, nt LB ei luba päringul teenuse poole pöörduda

##### Server.ServerProxy.ServiceFailed.MissingHeaderField  
Required field protocolVersion is missing

teenus tagastab päringu vastuse ilma protocolVersion elemendita

##### java.lang.OutOfMemoryError: Java heap space

[https://confluence.niis.org/pages/viewpage.action?pageId=4292877](https://confluence.niis.org/pages/viewpage.action?pageId=4292877)

##### Error 403 Forbidden

Teie turvaserverile ei ole ajatempliteenust avatud ehk kas puudub leping või ei ole teenusepakkuja lubanud Teie turvaserveri IP-d ehk puudub teenusele ligipääs.

##### Error 503

HTTP veakood 503 tähendab, et teenus ei ole kättesaadav.

##### Server.ServerProxy.ServiceFailed.InvalidContentType: Invalid content type: text/html

Teenus vastas turvaserverile mingi sodiga, st mitte xml. Teenusega on midagi valesti

##### Server.ServerProxy.ServiceFailed.HttpError, Server responded with error 404: Not Found

Tähendab, et teenus on maas, mingi vale seadistus teenuses, pole õige url äkki.

##### Server.ServerProxy.ServiceFailed.IOError: Read timed out

Teenus annab timeouti, st teenus ei vasta turvikule

##### Server responded with error 502: Bad Gateway

<div id="bkmrk-teenuse-poolne-viga."><div>Teenuse poolne viga. Kontrollida teenust ja selle toimimist</div></div>##### soap:Server Internal error

<div id="bkmrk-teenuse-poolne-viga">Teenuse poolne viga</div><div id="bkmrk-">  
</div>##### Server.ServerProxy.ServiceFailed.HttpError Server responded with error 302

teenuse endpoint url vale, redirectib valesse kohta LB-st. url vajab vb / lõppu

<div id="bkmrk--1">  
</div>

# Force restore conf backup

<div id="bkmrk-x-road-7-force-resto"><div>X-Road 7 force restore conf backup</div><div>Meant to be used on uninitialized ss and not in cluster</div>  
<div>From old ss</div><div>- take conf backup</div><div>- get UID from(UID is for ex. ee-dev/GOV/70009770/tehik_dev)</div></div>```
cat /etc/xroad/gpghome/openpgp-revocs.d/<-SOME-NUMBERS->.rev | grep uid
```

<div id="bkmrk-on-new-ss-slave%28s%29">  
<div>On new ss slave(s)</div></div>```
/usr/share/xroad/scripts/generate_gpg_keypair.sh /etc/xroad/gpghome <-UID->
```

<div id="bkmrk-on-new-ss-master">  
<div>On new ss master</div></div>```
/usr/share/xroad/scripts/generate_gpg_keypair.sh /etc/xroad/gpghome <-UID->
su xroad
/usr/share/xroad/scripts/restore_xroad_proxy_configuration.sh -f /tmp/ss-automatic-backup-2022_03_08_031501.gpg -F -N
```

<div id="bkmrk-create-cluster">  
<div>Create cluster</div></div><div id="bkmrk-">  
</div><div id="bkmrk-https%3A%2F%2Fgithub.com%2Fn">[https://github.com/nordic-institute/X-Road/blob/develop/doc/Manuals/ug-ss\_x-road\_6\_security\_server\_user\_guide.md#132-restore-from-the-command-line](https://github.com/nordic-institute/X-Road/blob/develop/doc/Manuals/ug-ss_x-road_6_security_server_user_guide.md#132-restore-from-the-command-line)</div>

# Increasing open files limit

<div id="bkmrk-">  
</div>##### Increasing xroad-proxy open file limits w/o host restart (temporary solution)

<div id="bkmrk-prlimit---pid-%3Cxroad">`prlimit --pid <xroad-proxy-PID> --nofile=10000:30000`</div><div id="bkmrk--1">  
</div>##### Increasing xroad-proxy open file limits

<div id="bkmrk-su---xroad">su - xroad</div><div id="bkmrk-prlimit---pid-%3Cxroad-1">prlimit --pid &lt;xroad-proxy-PID&gt;</div><div id="bkmrk--2">  
</div><div id="bkmrk-%2Fetc%2Fsecurity%2Flimits">/etc/security/limits.conf</div><div id="bkmrk-xroad-soft-nofile-10">xroad soft nofile 10000</div><div id="bkmrk-xroad-hard-nofile-30">xroad hard nofile 30000</div><div id="bkmrk--3">  
</div><div id="bkmrk-%2Flib%2Fsystemd%2Fsystem%2F">/lib/systemd/system/xroad-proxy.service</div><div id="bkmrk-limitnofile%3D10000">LimitNOFILE=10000</div><div id="bkmrk--4">  
</div><div id="bkmrk-systemctl-daemon-rel">systemctl daemon-reload</div><div id="bkmrk-service-xroad-proxy-">service xroad-proxy restart</div>

# Install specific version of xroad

List available versions and pick Your poison

```
`apt policy xroad-securityserver 2>/dev/null`
```

Export version as env, for ex

```
export CANDIDATE=7.3.2-1.ubuntu22.04
```

Install

```
apt-get install xroad-addon-hwtokens="$CANDIDATE" \
xroad-addon-messagelog="$CANDIDATE" \
xroad-addon-metaservices="$CANDIDATE" \
xroad-addon-opmonitoring="$CANDIDATE" \
xroad-addon-proxymonitor="$CANDIDATE" \
xroad-addon-wsdlvalidator="$CANDIDATE" \
xroad-base="$CANDIDATE" \
xroad-confclient="$CANDIDATE" \
xroad-database-local="$CANDIDATE" \
xroad-monitor="$CANDIDATE" \
xroad-opmonitor="$CANDIDATE" \
xroad-proxy="$CANDIDATE" \
xroad-proxy-ui-api="$CANDIDATE" \
xroad-securityserver="$CANDIDATE" \
xroad-securityserver-ee="$CANDIDATE" \
xroad-signer="$CANDIDATE"
```

# Links

X-tee kataloog - [https://x-tee.ee/catalogue](https://x-tee.ee/catalogue)

Info - [https://abi.ria.ee](https://abi.ria.ee/)

NIIS Github - [https://github.com/nordic-institute/X-Road](https://github.com/nordic-institute)

RIA Github X-Road scripts - [https://github.com/ria-ee/X-Road-scripts](https://github.com/ria-ee/X-Road-scripts)

Päringuid testimiseks - [https://abi.ria.ee/xtee/et/turvaserveri-haldus/soap-paeringud-testimiseks](https://abi.ria.ee/xtee/et/turvaserveri-haldus/soap-paeringud-testimiseks)

Epoch timestamp converter - [https://www.unixtimestamp.com](https://www.unixtimestamp.com/)

SK test certificate upload - [https://demo.sk.ee/upload\_cert/index.php](https://demo.sk.ee/upload_cert/index.php)

# Messagelog Database info

[https://moodle.ria.ee/mod/page/view.php?id=694](https://moodle.ria.ee/mod/page/view.php?id=694)

```
psql -h 127.0.0.1 -U messagelog <password from /etc/xroad/db.properties>
```

##### Kui palju kirjeid on tabelis

```
select count( *) from logrecord;
```

##### Milline on kõige vanem sõnumilogi kirje baasis

```
select to_timestamp(min( time )::float/1000) from logrecord;
```

##### Kui palju on ajatembedamata sõnumeid

```
select count(1) from logrecord where discriminator::text = 'm'::text and signaturehash is not null;
```

##### Viimane ajatembeldamata sõnum

```
select to_timestamp(min(time)::float/1000) from logrecord where discriminator::text = 'm'::text and signaturehash is not null;
```

##### Kui palju on ajatembeldatud aga arhiveerimata sõnumeid

```
select count(1) from logrecord where timestamprecord in (select id from logrecord where discriminator::text = 't'::text and archived = false);
```

##### Viimane ajatembeldatud aga arhiveerimata sõnum

```
select to_timestamp(min(time)::float/1000) from logrecord where timestamprecord in (select id from logrecord where discriminator::text = 't'::text and archived = false);
```

##### Size of the largest row in logrecord table

```
select t.id, t.archived, (pg_column_size(t.message)) as size from logrecord t where message IS NOT NULL order by size desc;
```

# MISP2

##### If base package is upgraded, port 8080 conf will be defaulted

Uncomment port 8080 in `/var/lib/tomcat8/conf/server.xml`

##### Remote database upgrade

```
psql -h <database-ip> -p <database-port> -U postgres -d misp2db -f insert_xslt.sql
```

##### When was MISP2 upgraded to newer version

```
ls -t /var/log/dpkg* | xargs zgrep "upgrade xtee-misp2-application"
```

##### Admintool needs to be run as admin

`/usr/xtee/app/admintool.sh`

##### Conf

`/var/lib/tomcat8/webapps/misp2/WEB-INF/classes/config.cfg`

##### java.lang.OutOfMemoryError: Java heap space - increasing memory

Open /etc/default/tomcat8 and increase MaxPermSize=

misp apteegikood tuleb rets teenusest. add/edit = tervisekassa  
Apteekrikoodi registreerimiseks peab pöörduma Tervisekassa poole, kes haldab 'rets' andmekogu teenuseid.

# Op-mon to use one IP and daemon within a cluster

1\) make sure, slave nodes -&gt; master node can connect over port 2080

2\) on master node, add to /etc/xroad/conf.d/local.ini

```
[op-monitor]
host = <master-internal-IP>
```

<div id="bkmrk-3%29-on-master-node"><div>3) on master node</div></div><div id="bkmrk-service-xroad-proxy-">`service xroad-proxy restart`</div><div id="bkmrk-4%29-on-slave-nodes">4) on slave nodes</div><div id="bkmrk-service-xroad-proxy--1">`service xroad-proxy restart`</div><div id="bkmrk-service-xroad-opmon-"><div><div>`service xroad-opmon stop`</div><div>`service xroad-opmon mask`</div></div></div>

# REST examples

##### listMethods

```
curl -X GET -H "accept: application/json" -H "X-Road-Client: ee-dev/GOV/70008799/pohak" "http://10.0.13.90/r1/ee-dev/GOV/70008799/pohak/listMethods" | json_pp 
```

##### HTTPS with cert and key

```
curl --cert nextcloud.bgp.12.berylia.org_cert.crt --key nextcloud.bgp.12.berylia.org_key.crt -X GET -H "accept: application/json" -H "X-Road-Client:BERYLIA/GOV/1003/recon" "https://xroad-securityserver.bgp.12.berylia.org/r1/BERYLIA/GOV/1002/satellite/listMethods" | json_pp
```

##### getOpenAPI

```
curl -k -X GET -H "accept: application/json" -H "X-Road-Client: ee-dev/GOV/70008799/pohak" "https://10.0.13.90/r1/ee-dev/GOV/70008799/pohak/getOpenAPI?serviceCode=adverse-event" | json_pp
```

##### POST

```
curl -k -X POST "https://10.0.14.26/r1/ee-test/GOV/70001969/tookeskk/digitaalne_teatis/api/v1/digitaalne_teatis" -H "accept: application/json" -H "Content-Type: application/json" -H "X-Road-Client: ee-test/GOV/70001969/tookeskk" \
--data '{
    "noticeId": "",
    "employeeIdCode": "",
    "employeeFirstName": "",
    "employeeLastName": "",
    "employeePhone": ""
}'
```

More help: [https://x-tee.ee/docs/live/xroad/pr-rest\_x-road\_message\_protocol\_for\_rest.html#64-post-request-and-response](https://x-tee.ee/docs/live/xroad/pr-rest_x-road_message_protocol_for_rest.html#64-post-request-and-response)

# rm -rf

```
apt purge --remove -y xroad-addon-hwtokens xroad-addon-messagelog   xroad-addon-metaservices xroad-addon-opmonitoring xroad-addon-proxymonitor xroad-addon-wsdlvalidator xroad-base               xroad-confclient         xroad-database-local     xroad-monitor            xroad-opmonitor          xroad-proxy              xroad-proxy-ui-api       xroad-securityserver-ee xroad-securityserver  xroad-signer

apt purge --remove -y postgresql postgresql-10 postgresql-client-10 postgresql-client-common postgresql-common postgresql-contrib

apt purge --remove -y rsyslog auditd && rm -rf /etc/rsyslog.d/* && rm -rf /etc/audit/rules.d/* 

apt autoclean -y && apt autoremove -y

userdel -r xroad
userdel -r xroad-slave
rm -rf /etc/xroad /var/lib/xroad /etc/xroad /etc/xroad.properties /usr/share/xroad /var/lib/xroad /var/log/xroad /var/tmp/xroad /etc/cron.d/xroad* /etc/systemd/system/xroad* /etc/zabbix/zabbix_agent2.d/userparameter_xroad* /etc/logrotate.d/xroad*
```

# Testpäring

```
<?xml version="1.0" encoding="utf-8"?>
<SOAP-ENV:Envelope
        xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/"
        xmlns:xroad="http://x-road.eu/xsd/xroad.xsd"
        xmlns:id="http://x-road.eu/xsd/identifiers">
    <SOAP-ENV:Header>
        <xroad:client id:objectType="SUBSYSTEM">
            <id:xRoadInstance>EE</id:xRoadInstance>
            <id:memberClass>GOV</id:memberClass>
            <id:memberCode>70009770</id:memberCode>
            <id:subsystemCode>digilugu</id:subsystemCode>
        </xroad:client>
        <xroad:service id:objectType="SERVICE">
         <id:xRoadInstance>EE</id:xRoadInstance>
         <id:memberClass>GOV</id:memberClass>
         <id:memberCode>70009770</id:memberCode>
         <id:subsystemCode>digilugu</id:subsystemCode>
         <id:serviceCode>listMethods</id:serviceCode>
        </xroad:service>
        <xroad:id>{{v4uuid}}</xroad:id>
        <xroad:protocolVersion>4.0</xroad:protocolVersion>
    </SOAP-ENV:Header>
    <SOAP-ENV:Body>
        <xroad:listMethods/>
    </SOAP-ENV:Body>
</SOAP-ENV:Envelope>
```

```
curl -v -d @req.xml -o resp.xml -H "Content-type: text/xml; charset=UTF-8" http://TURVASERVER
```

Peale päringu tegemist piiluda resp.xml sisse, et näha päringu vastust

# Timeouts

client-httpclient-timeout default=0  
The maximum time (SO\_TIMEOUT, in milliseconds) that connections from a service consuming security server to a service providing security server are allowed to wait for a response before the consumer end httpclient gives up. Value of 0 means that an infinite wait time is allowed

ma ütleks, et vaikimisi see polegi piiratud

kui turvikud loovad omavahel ühenduse, siis oodatakse seni, kuni üks pool loobub (timeouti pole). Vaikimisi producer ootab producer IS vastust 60sekundit, mida saab muuta producer admin UI-st

# Tips & Tricks

<div id="bkmrk-"><div><div>  
</div></div></div>##### When was xroad-securityserver package last upgraded

```
ls -t /var/log/dpkg* | xargs zgrep "upgrade xroad-securityserver"
```

##### Editing keyconf

1. On master node, stop xroad-signer process
2. Edit keyconf
3. On master node, start xroad-signer process
4. Restart xroad-signer on slave nodes

##### Grep packet loss

```
cat /var/log/xroad/*.log|grep -v "org.eclipse.jetty.io.EofException: null"|wc -l
```

##### Time opmonitor database read

```
time echo "select count(id) from operational_data;" | psql -h 127.0.0.1 -U opmonitor -W op-monitor
```

##### Keyconfist välja parseda serdid normaliseeritud kujul

```
cat /etc/xroad/signer/keyconf.xml | python -c "exec(\"import sys, xml.etree.ElementTree as ET\\nroot=ET.fromstring(sys.stdin.read())\\nfor cert in root.findall('.//cert/contents'):\\n print '-----BEGIN CERTIFICATE-----\\\\n'+str('\\\\n').join([cert.text[i:i+64] for i in range(0, len(cert.text), 64)])+'\\\\n-----END CERTIFICATE-----'\")"
```

##### Logback location

```
/etc/xroad/conf.d/
```

##### Global conf location in ss

```
/etc/xroad/globalconf/<environment>/shared-params.xml
```

##### List xroad services status

```
`systemctl list-units "xroad*"`
```

##### Stop all xroad processes

```
systemctl stop xroad-*
```

##### Environmental monitoring

is the monitoring of the X-Road environment: details of the security servers such as operating system, memory, disk space, CPU load, running processes and installed packages, etc.

##### Operational monitoring

is the monitoring of operational statistics such as which services have been called, how many times, what is the average response time, etc.

##### Count proxy open files

```
lsof -p `systemctl show -p MainPID xroad-proxy.service|cut -d'=' -f2`|wc -l
```

# Upgrade procedure

[https://www.x-tee.ee/docs/live/xroad/ig-xlb\_x-road\_external\_load\_balancer\_installation\_guide.html#72-online-rolling-upgrade](https://www.x-tee.ee/docs/live/xroad/ig-xlb_x-road_external_load_balancer_installation_guide.html#72-online-rolling-upgrade)

Enable maintenance mode(run from security server)

`curl <a href="http://localhost:5566/maintenance?targetState=false">http://localhost:5566/maintenance?targetState=true</a>`

Watch connections and wait until all requests are finished

`watch -n1 ss -tn state established sport = :5500 or dport = :5500`

Make snapshot of machine

Unhold xroad packages

`apt-mark unhold xroad-*`

Update package list

`apt update`

Upgrade xroad packages

`apt install xroad-securityserver-ee`

Hold xroad packages

`apt-mark hold xroad-*`

Disable maintanance mode(run from security server)

`curl <a href="http://localhost:5566/maintenance?targetState=false">http://localhost:5566/maintenance?targetState=false</a>`

# WSDL validator

From Security Server CLI

```
/usr/share/xroad/wsdlvalidator/bin/wsdlvalidator_wrapper.sh tam6.wsdl
```

```
/usr/share/xroad/wsdlvalidator/bin/wsdlvalidator_wrapper.sh http://10.13.24.14/adapter/tam6.wsdl
```

Security Server uses Apache CXF wsdlvalidator

[https://github.com/nordic-institute/X-Road/blob/develop/src/addons/wsdlvalidator/build.gradle#L22](https://github.com/nordic-institute/X-Road/blob/develop/src/addons/wsdlvalidator/build.gradle#L22)

[https://cxf.apache.org/docs/wsdlvalidator.html](https://cxf.apache.org/docs/wsdlvalidator.html)

[https://github.com/nordic-institute/wsdlvalidator](https://github.com/nordic-institute/wsdlvalidator)

# X-tee kataloog

##### Viga OpenAPI kirjelduse laadimisel või töötlemisel

Kontrollida üle "content-type HTTP header". Peaks vastama openapi kirjelduse formaadile, kas siis json või yaml, nt “application/json” või “application/yaml”

##### Kataloogi uuendatakse iga 1h tagant